[==========] Running 38 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 37 tests from ExternalMiniClusterTest
[ RUN ] ExternalMiniClusterTest.TestKerberosReacquire
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 01:49:41 dist-test-slave-wg14 krb5kdc[20493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 01:49:41 dist-test-slave-wg14 krb5kdc[20493](info): set up 2 sockets
Jun 20 01:49:41 dist-test-slave-wg14 krb5kdc[20493](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 01:49:43.329437 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.061s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 01:49:43 dist-test-slave-wg14 krb5kdc[20493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T01:49:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:49:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:49:43.597604 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:41997
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:35433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:41997 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:49:44.073715 20509 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:49:44.074167 20509 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:49:44.074318 20509 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:49:44.084146 20509 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:49:44.084300 20509 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:49:44.084383 20509 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:49:44.084456 20509 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:49:44.097909 20509 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:35433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:41997
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:41997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:49:44.099941 20509 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:49:44.101716 20509 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:49:44.114427 20514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:49:44.114573 20515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:49:44.114573 20517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:49:44.115845 20509 server_base.cc:1061] running on GCE node
I20260620 01:49:44.117230 20509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:49:44.120400 20509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:49:44.121935 20509 hybrid_clock.cc:648] HybridClock initialized: now 1781920184121768 us; error 94 us; skew 500 ppm
I20260620 01:49:44.122704 20509 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:49:44 dist-test-slave-wg14 krb5kdc[20493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920184, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:49:44.128760 20509 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:49:44.132550 20509 webserver.cc:533] Webserver started at http://127.20.1.190:40981/ using document root <none> and password file <none>
I20260620 01:49:44.134052 20509 fs_manager.cc:362] Metadata directory not provided
I20260620 01:49:44.134351 20509 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:49:44.134999 20509 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:49:44.139621 20509 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "1a09c9a877f345d089fa66afcc5c8c23"
format_stamp: "Formatted at 2026-06-20 01:49:44 on dist-test-slave-wg14"
I20260620 01:49:44.140923 20509 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "1a09c9a877f345d089fa66afcc5c8c23"
format_stamp: "Formatted at 2026-06-20 01:49:44 on dist-test-slave-wg14"
I20260620 01:49:44.150128 20509 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.001s sys 0.007s
I20260620 01:49:44.156975 20524 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:49:44.159927 20509 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:49:44.160223 20509 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "1a09c9a877f345d089fa66afcc5c8c23"
format_stamp: "Formatted at 2026-06-20 01:49:44 on dist-test-slave-wg14"
I20260620 01:49:44.160564 20509 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:49:44.205353 20509 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:49:44.217690 20509 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:49:44.218604 20509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:49:44.253531 20509 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:41997
I20260620 01:49:44.253572 20575 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:41997 every 8 connection(s)
I20260620 01:49:44.255916 20509 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:49:44.262758 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 20509
I20260620 01:49:44.263203 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:49:44.263810 20576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:49:44.280359 20576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23: Bootstrap starting.
I20260620 01:49:44.284960 20576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23: Neither blocks nor log segments found. Creating new log.
I20260620 01:49:44.286590 20576 log.cc:826] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23: Log is configured to *not* fsync() on all Append() calls
I20260620 01:49:44.294574 20576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23: No bootstrap required, opened a new log
I20260620 01:49:44.305159 20576 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09c9a877f345d089fa66afcc5c8c23" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41997 } }
I20260620 01:49:44.306206 20576 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:49:44.306440 20576 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a09c9a877f345d089fa66afcc5c8c23, State: Initialized, Role: FOLLOWER
I20260620 01:49:44.308032 20576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09c9a877f345d089fa66afcc5c8c23" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41997 } }
I20260620 01:49:44.308709 20576 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 01:49:44.309206 20576 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 01:49:44.309645 20576 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 01:49:44 dist-test-slave-wg14 krb5kdc[20493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:49:44.314038 20576 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09c9a877f345d089fa66afcc5c8c23" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41997 } }
I20260620 01:49:44.315248 20576 leader_election.cc:304] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a09c9a877f345d089fa66afcc5c8c23; no voters:
I20260620 01:49:44.316666 20576 leader_election.cc:290] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 01:49:44.317094 20581 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:49:44.319948 20581 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [term 1 LEADER]: Becoming Leader. State: Replica: 1a09c9a877f345d089fa66afcc5c8c23, State: Running, Role: LEADER
I20260620 01:49:44.320895 20576 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:49:44.321141 20581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09c9a877f345d089fa66afcc5c8c23" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41997 } }
I20260620 01:49:44.327488 20582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a09c9a877f345d089fa66afcc5c8c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09c9a877f345d089fa66afcc5c8c23" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41997 } } }
I20260620 01:49:44.328085 20582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [sys.catalog]: This master's current role is: LEADER
I20260620 01:49:44.330888 20583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a09c9a877f345d089fa66afcc5c8c23. Latest consensus state: current_term: 1 leader_uuid: "1a09c9a877f345d089fa66afcc5c8c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09c9a877f345d089fa66afcc5c8c23" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41997 } } }
I20260620 01:49:44.333425 20583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [sys.catalog]: This master's current role is: LEADER
I20260620 01:49:44.339519 20591 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:49:44.348059 20591 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:49:44.365738 20591 catalog_manager.cc:1382] Generated new cluster ID: c87ae97977d34ab5b120e9b036fc0e44
I20260620 01:49:44.365973 20591 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:49:44.402853 20591 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:49:44.405372 20591 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:49:44.422140 20591 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23: Generated new TSK 0
I20260620 01:49:44.424930 20591 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:49:44.585026 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:41997
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:35433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:49:45.157392 20604 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:49:45.157783 20604 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:49:45.157912 20604 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:49:45.170609 20604 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:49:45.170876 20604 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:49:45.184513 20604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:35433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:41997
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:49:45.186635 20604 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:49:45.189321 20604 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:49:45.211210 20610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:49:45.211344 20609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:49:45.212306 20612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:49:45.212699 20604 server_base.cc:1061] running on GCE node
I20260620 01:49:45.213861 20604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:49:45.215469 20604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:49:45.219072 20604 hybrid_clock.cc:648] HybridClock initialized: now 1781920185218828 us; error 210 us; skew 500 ppm
I20260620 01:49:45.220049 20604 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:49:45 dist-test-slave-wg14 krb5kdc[20493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920185, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:49:45.226854 20604 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:49:45.229737 20604 webserver.cc:533] Webserver started at http://127.20.1.129:43817/ using document root <none> and password file <none>
I20260620 01:49:45.230999 20604 fs_manager.cc:362] Metadata directory not provided
I20260620 01:49:45.231257 20604 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:49:45.231734 20604 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:49:45.235929 20604 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "b7e5817da34e44be87848d3bc6903ea0"
format_stamp: "Formatted at 2026-06-20 01:49:45 on dist-test-slave-wg14"
I20260620 01:49:45.237182 20604 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "b7e5817da34e44be87848d3bc6903ea0"
format_stamp: "Formatted at 2026-06-20 01:49:45 on dist-test-slave-wg14"
I20260620 01:49:45.246604 20604 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260620 01:49:45.252326 20619 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:49:45.255092 20604 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:49:45.255405 20604 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "b7e5817da34e44be87848d3bc6903ea0"
format_stamp: "Formatted at 2026-06-20 01:49:45 on dist-test-slave-wg14"
I20260620 01:49:45.255663 20604 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:49:45.279457 20604 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:49:45.287573 20604 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:49:45.287923 20604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:49:45.289263 20604 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:49:45.291790 20604 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:49:45.291952 20604 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:49:45.292058 20604 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:49:45.292095 20604 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:49:45.351033 20604 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38863
I20260620 01:49:45.351086 20731 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38863 every 8 connection(s)
I20260620 01:49:45.352870 20604 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:49:45.361397 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 20604
I20260620 01:49:45.361953 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 01:49:45 dist-test-slave-wg14 krb5kdc[20493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920185, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:49:45.386564 20732 heartbeater.cc:344] Connected to a master server at 127.20.1.190:41997
I20260620 01:49:45.387048 20732 heartbeater.cc:461] Registering TS with master...
I20260620 01:49:45.388231 20732 heartbeater.cc:507] Master 127.20.1.190:41997 requested a full tablet report, sending...
I20260620 01:49:45.391587 20541 ts_manager.cc:194] Registered new tserver with Master: b7e5817da34e44be87848d3bc6903ea0 (127.20.1.129:38863)
I20260620 01:49:45.393721 20541 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:40317
I20260620 01:49:45.398571 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:49:46.398734 20732 heartbeater.cc:499] Master 127.20.1.190:41997 was elected leader, sending a full tablet report...
Jun 20 01:49:59 dist-test-slave-wg14 krb5kdc[20493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920199, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:49:59.131778 20520 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 01:50:00 dist-test-slave-wg14 krb5kdc[20493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920200, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:50:00.230307 20615 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 01:50:01 dist-test-slave-wg14 krb5kdc[20493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260620 01:50:01.433542 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 20509
I20260620 01:50:01.435882 20571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:01.568632 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 20509
I20260620 01:50:01.591804 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:41997
--webserver_interface=127.20.1.190
--webserver_port=40981
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:35433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:41997 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:50:02.120863 20739 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:02.121296 20739 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:02.121448 20739 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:02.133060 20739 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:02.133210 20739 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:02.133270 20739 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:02.133312 20739 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:02.149503 20739 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:35433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:41997
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:41997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=40981
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:02.151496 20739 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:02.153187 20739 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:02.168169 20745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:02.168414 20744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:02.169559 20739 server_base.cc:1061] running on GCE node
W20260620 01:50:02.169762 20747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:02.170869 20739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:02.173316 20739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:02.174827 20739 hybrid_clock.cc:648] HybridClock initialized: now 1781920202174716 us; error 75 us; skew 500 ppm
I20260620 01:50:02.175837 20739 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:50:02 dist-test-slave-wg14 krb5kdc[20493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920202, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:50:02.181326 20739 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:50:02.183816 20739 webserver.cc:533] Webserver started at http://127.20.1.190:40981/ using document root <none> and password file <none>
I20260620 01:50:02.184727 20739 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:02.184882 20739 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:02.192467 20739 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 01:50:02.197160 20754 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:02.200263 20739 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260620 01:50:02.200558 20739 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "1a09c9a877f345d089fa66afcc5c8c23"
format_stamp: "Formatted at 2026-06-20 01:49:44 on dist-test-slave-wg14"
I20260620 01:50:02.201675 20739 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:02.226507 20739 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:02.232050 20739 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:02.232422 20739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:02.260746 20739 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:41997
I20260620 01:50:02.260911 20805 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:41997 every 8 connection(s)
I20260620 01:50:02.265847 20739 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:50:02.272166 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 20739
I20260620 01:50:02.277854 20806 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:50:02.283365 20806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23: Bootstrap starting.
Jun 20 01:50:02 dist-test-slave-wg14 krb5kdc[20493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:50:02.309703 20806 log.cc:826] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:02.326537 20806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:50:02.327322 20806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23: Bootstrap complete.
I20260620 01:50:02.336930 20806 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09c9a877f345d089fa66afcc5c8c23" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41997 } }
I20260620 01:50:02.338335 20806 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a09c9a877f345d089fa66afcc5c8c23, State: Initialized, Role: FOLLOWER
I20260620 01:50:02.339493 20806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09c9a877f345d089fa66afcc5c8c23" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41997 } }
I20260620 01:50:02.339843 20806 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 01:50:02.339984 20806 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 01:50:02.340214 20806 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:50:02.344573 20806 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09c9a877f345d089fa66afcc5c8c23" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41997 } }
I20260620 01:50:02.346001 20806 leader_election.cc:304] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a09c9a877f345d089fa66afcc5c8c23; no voters:
I20260620 01:50:02.347086 20806 leader_election.cc:290] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 01:50:02.347463 20811 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 01:50:02.348482 20811 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [term 2 LEADER]: Becoming Leader. State: Replica: 1a09c9a877f345d089fa66afcc5c8c23, State: Running, Role: LEADER
I20260620 01:50:02.349184 20811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09c9a877f345d089fa66afcc5c8c23" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41997 } }
I20260620 01:50:02.349934 20806 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:02.352159 20812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1a09c9a877f345d089fa66afcc5c8c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09c9a877f345d089fa66afcc5c8c23" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41997 } } }
I20260620 01:50:02.352682 20812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:02.355468 20813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a09c9a877f345d089fa66afcc5c8c23. Latest consensus state: current_term: 2 leader_uuid: "1a09c9a877f345d089fa66afcc5c8c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09c9a877f345d089fa66afcc5c8c23" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41997 } } }
I20260620 01:50:02.356276 20813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:02.361143 20818 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:50:02.375943 20818 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:50:02.378222 20818 catalog_manager.cc:1294] Loaded cluster ID: c87ae97977d34ab5b120e9b036fc0e44
I20260620 01:50:02.378536 20818 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:50:02.387338 20818 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:50:02.390269 20818 catalog_manager.cc:6080] T 00000000000000000000000000000000 P 1a09c9a877f345d089fa66afcc5c8c23: Loaded TSK: 0
I20260620 01:50:02.393033 20818 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:50:02.594765 20771 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b7e5817da34e44be87848d3bc6903ea0" instance_seqno: 1781920185333433) as {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:49497; Asking this server to re-register.
I20260620 01:50:02.596762 20732 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:02.597319 20732 heartbeater.cc:507] Master 127.20.1.190:41997 requested a full tablet report, sending...
I20260620 01:50:02.601744 20771 ts_manager.cc:194] Registered new tserver with Master: b7e5817da34e44be87848d3bc6903ea0 (127.20.1.129:38863)
I20260620 01:50:02.605264 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:50:02.605789 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 20604
Jun 20 01:50:02 dist-test-slave-wg14 krb5kdc[20493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 01:50:02.638140 20727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:02.768915 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 20604
I20260620 01:50:02.801271 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 20739
I20260620 01:50:02.803108 20801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:02.935092 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 20739
2026-06-20T01:50:02Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestKerberosReacquire (21728 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/0
I20260620 01:50:02.980747 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:46541
--webserver_interface=127.20.1.190
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117 with env {}
W20260620 01:50:03.522976 20834 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:03.523463 20834 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:03.523586 20834 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:03.535468 20834 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:03.535938 20834 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:03.536139 20834 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:03.536216 20834 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:03.549293 20834 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:46541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:03.552999 20834 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:03.556397 20834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:03.568565 20839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:03.569324 20840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:03.570363 20842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:03.571942 20834 server_base.cc:1061] running on GCE node
I20260620 01:50:03.573522 20834 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:03.573781 20834 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:03.573879 20834 hybrid_clock.cc:648] HybridClock initialized: now 1781920203573840 us; error 0 us; skew 500 ppm
I20260620 01:50:03.574321 20834 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:03.579447 20834 webserver.cc:533] Webserver started at http://127.20.1.190:44931/ using document root <none> and password file <none>
I20260620 01:50:03.580482 20834 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:03.580627 20834 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:03.581007 20834 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:03.583889 20834 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "d7d05e51ca25403e993415676dcaaf6e"
format_stamp: "Formatted at 2026-06-20 01:50:03 on dist-test-slave-wg14"
I20260620 01:50:03.584939 20834 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "d7d05e51ca25403e993415676dcaaf6e"
format_stamp: "Formatted at 2026-06-20 01:50:03 on dist-test-slave-wg14"
I20260620 01:50:03.595696 20834 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s
I20260620 01:50:03.602699 20847 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:03.605948 20834 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260620 01:50:03.606292 20834 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "d7d05e51ca25403e993415676dcaaf6e"
format_stamp: "Formatted at 2026-06-20 01:50:03 on dist-test-slave-wg14"
I20260620 01:50:03.606616 20834 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:03.652479 20834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:03.654011 20834 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:03.654348 20834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:03.687815 20834 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:46541
I20260620 01:50:03.687624 20898 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:46541 every 8 connection(s)
I20260620 01:50:03.690989 20834 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:50:03.696913 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 20834
I20260620 01:50:03.697290 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:50:03.698144 20899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:50:03.700048 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:44845
--webserver_interface=127.20.1.189
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117 with env {}
I20260620 01:50:03.711167 20899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:03.751379 20899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:50:03.753856 20851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:44845: connect: Connection refused (error 111)
W20260620 01:50:03.757439 20899 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:44845: Network error: Client connection negotiation failed: client connection to 127.20.1.189:44845: connect: Connection refused (error 111)
I20260620 01:50:03.807085 20899 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } attempt: 1
W20260620 01:50:03.810623 20899 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:44845: Network error: Client connection negotiation failed: client connection to 127.20.1.189:44845: connect: Connection refused (error 111)
I20260620 01:50:03.879175 20899 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } attempt: 2
W20260620 01:50:03.884019 20899 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:44845: Network error: Client connection negotiation failed: client connection to 127.20.1.189:44845: connect: Connection refused (error 111)
I20260620 01:50:03.975348 20899 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } attempt: 3
W20260620 01:50:03.978076 20899 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:44845: Network error: Client connection negotiation failed: client connection to 127.20.1.189:44845: connect: Connection refused (error 111)
I20260620 01:50:04.121501 20899 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } attempt: 4
W20260620 01:50:04.125651 20899 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:44845: Network error: Client connection negotiation failed: client connection to 127.20.1.189:44845: connect: Connection refused (error 111)
W20260620 01:50:04.220939 20901 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:04.221439 20901 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:04.221550 20901 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:04.233865 20901 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:04.234097 20901 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:04.234185 20901 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:04.234263 20901 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:04.250319 20901 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:44845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:04.253871 20901 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:04.258442 20901 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:04.272749 20909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:04.272562 20910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:04.274655 20901 server_base.cc:1061] running on GCE node
W20260620 01:50:04.275628 20912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:04.277062 20901 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:04.277321 20901 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:04.277432 20901 hybrid_clock.cc:648] HybridClock initialized: now 1781920204277399 us; error 0 us; skew 500 ppm
I20260620 01:50:04.277853 20901 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:04.283425 20901 webserver.cc:533] Webserver started at http://127.20.1.189:36673/ using document root <none> and password file <none>
I20260620 01:50:04.285110 20901 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:04.285355 20901 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:04.286963 20901 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:04.293040 20901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "4f7675b441a14937960e9dbe0cae4ea8"
format_stamp: "Formatted at 2026-06-20 01:50:04 on dist-test-slave-wg14"
I20260620 01:50:04.294184 20901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "4f7675b441a14937960e9dbe0cae4ea8"
format_stamp: "Formatted at 2026-06-20 01:50:04 on dist-test-slave-wg14"
I20260620 01:50:04.305819 20901 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.011s sys 0.001s
I20260620 01:50:04.311957 20917 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:04.314368 20901 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260620 01:50:04.314694 20901 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "4f7675b441a14937960e9dbe0cae4ea8"
format_stamp: "Formatted at 2026-06-20 01:50:04 on dist-test-slave-wg14"
I20260620 01:50:04.315047 20901 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:04.345520 20901 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:04.348745 20901 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:04.349169 20901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:04.386078 20901 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:44845
I20260620 01:50:04.386106 20968 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:44845 every 8 connection(s)
I20260620 01:50:04.389166 20901 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:50:04.390728 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 20901
I20260620 01:50:04.391031 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:50:04.392411 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:44117
--webserver_interface=127.20.1.188
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117 with env {}
I20260620 01:50:04.397486 20969 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:50:04.410732 20969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:04.425277 20899 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } attempt: 5
I20260620 01:50:04.441447 20969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:04.450573 20899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:50:04.455250 20899 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44117: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44117: connect: Connection refused (error 111)
I20260620 01:50:04.460698 20969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:50:04.464826 20918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44117: connect: Connection refused (error 111)
W20260620 01:50:04.469939 20969 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44117: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44117: connect: Connection refused (error 111)
I20260620 01:50:04.506772 20899 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } attempt: 1
W20260620 01:50:04.510517 20899 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44117: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44117: connect: Connection refused (error 111)
I20260620 01:50:04.520166 20969 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } attempt: 1
W20260620 01:50:04.523048 20969 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44117: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44117: connect: Connection refused (error 111)
I20260620 01:50:04.578866 20899 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } attempt: 2
W20260620 01:50:04.583655 20899 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44117: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44117: connect: Connection refused (error 111)
I20260620 01:50:04.591498 20969 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } attempt: 2
W20260620 01:50:04.596156 20969 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44117: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44117: connect: Connection refused (error 111)
I20260620 01:50:04.687788 20969 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } attempt: 3
I20260620 01:50:04.689996 20899 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } attempt: 3
W20260620 01:50:04.691493 20969 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44117: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44117: connect: Connection refused (error 111)
W20260620 01:50:04.692958 20899 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44117: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44117: connect: Connection refused (error 111)
I20260620 01:50:04.834928 20969 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } attempt: 4
W20260620 01:50:04.837772 20969 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44117: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44117: connect: Connection refused (error 111)
I20260620 01:50:04.870275 20899 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } attempt: 4
W20260620 01:50:04.874579 20899 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44117: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44117: connect: Connection refused (error 111)
W20260620 01:50:04.907779 20971 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:04.908265 20971 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:04.908385 20971 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:04.919749 20971 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:04.919890 20971 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:04.919973 20971 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:04.920050 20971 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:04.933423 20971 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:44117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:04.935491 20971 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:04.937814 20971 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:04.953878 20984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:04.954146 20982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:04.954056 20981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:04.955833 20971 server_base.cc:1061] running on GCE node
I20260620 01:50:04.956862 20971 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:04.957079 20971 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:04.957178 20971 hybrid_clock.cc:648] HybridClock initialized: now 1781920204957148 us; error 0 us; skew 500 ppm
I20260620 01:50:04.957616 20971 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:04.963806 20971 webserver.cc:533] Webserver started at http://127.20.1.188:37181/ using document root <none> and password file <none>
I20260620 01:50:04.965284 20971 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:04.965534 20971 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:04.965976 20971 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:04.969341 20971 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "baaadc5c42084ad8b930708422b8fda6"
format_stamp: "Formatted at 2026-06-20 01:50:04 on dist-test-slave-wg14"
I20260620 01:50:04.970597 20971 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "baaadc5c42084ad8b930708422b8fda6"
format_stamp: "Formatted at 2026-06-20 01:50:04 on dist-test-slave-wg14"
I20260620 01:50:04.979656 20971 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.004s
I20260620 01:50:04.989079 20989 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:04.992148 20971 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.003s sys 0.003s
I20260620 01:50:04.992417 20971 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "baaadc5c42084ad8b930708422b8fda6"
format_stamp: "Formatted at 2026-06-20 01:50:04 on dist-test-slave-wg14"
I20260620 01:50:04.992702 20971 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:05.031769 20971 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:05.033507 20971 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:05.034013 20971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:05.076496 21040 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:44117 every 8 connection(s)
I20260620 01:50:05.076522 20971 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:44117
I20260620 01:50:05.080268 20971 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:50:05.084870 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 20971
I20260620 01:50:05.085099 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:50:05.091616 21041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:50:05.105348 21041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:05.129331 21041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:05.137164 20969 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } attempt: 5
I20260620 01:50:05.142299 21041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:05.151870 20899 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } attempt: 5
I20260620 01:50:05.155395 20969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8: Bootstrap starting.
I20260620 01:50:05.161655 20969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8: Neither blocks nor log segments found. Creating new log.
I20260620 01:50:05.163971 20969 log.cc:826] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:05.169520 21041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6: Bootstrap starting.
I20260620 01:50:05.169659 20969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8: No bootstrap required, opened a new log
I20260620 01:50:05.170763 20899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: Bootstrap starting.
I20260620 01:50:05.174927 21041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6: Neither blocks nor log segments found. Creating new log.
I20260620 01:50:05.175262 20899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: Neither blocks nor log segments found. Creating new log.
I20260620 01:50:05.176443 20899 log.cc:826] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:05.176512 21041 log.cc:826] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:05.177728 20969 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:05.178138 20969 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:50:05.178267 20969 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7675b441a14937960e9dbe0cae4ea8, State: Initialized, Role: FOLLOWER
I20260620 01:50:05.180487 20899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: No bootstrap required, opened a new log
I20260620 01:50:05.180874 21041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6: No bootstrap required, opened a new log
I20260620 01:50:05.181699 20969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:05.190675 21052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:05.191105 21052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:05.191498 21041 raft_consensus.cc:359] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:05.192102 21041 raft_consensus.cc:385] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:50:05.192284 21041 raft_consensus.cc:740] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baaadc5c42084ad8b930708422b8fda6, State: Initialized, Role: FOLLOWER
I20260620 01:50:05.193446 20969 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:05.193276 21041 consensus_queue.cc:260] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:05.193624 20899 raft_consensus.cc:359] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:05.194058 20899 raft_consensus.cc:385] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:50:05.194243 20899 raft_consensus.cc:740] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d05e51ca25403e993415676dcaaf6e, State: Initialized, Role: FOLLOWER
I20260620 01:50:05.194803 21053 sys_catalog.cc:455] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:05.195652 21053 sys_catalog.cc:458] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:05.195595 20899 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:05.198199 21041 sys_catalog.cc:565] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:05.198144 21055 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:05.199314 21055 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:05.200631 20899 sys_catalog.cc:565] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:05.252307 20901 master_runner.cc:187] Error getting master registration for 127.20.1.190:46541: OK, instance_id { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" instance_seqno: 1781920203676718 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:50:05.268077 20901 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:50:05.272008 21080 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:50:05.272228 21080 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 01:50:05.288849 20995 tablet.cc:2406] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:50:05.276039 20971 master_runner.cc:187] Error getting master registration for 127.20.1.190:46541: OK, instance_id { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" instance_seqno: 1781920203676718 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:50:05.292099 20971 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:50:05.292292 21086 catalog_manager.cc:1593] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:50:05.292807 21086 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:50:05.319051 20834 master_runner.cc:264] No leader master found from master d7d05e51ca25403e993415676dcaaf6e
I20260620 01:50:05.319278 20834 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:50:05.322858 21089 catalog_manager.cc:1593] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:50:05.323251 21089 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 01:50:05.353040 20923 tablet.cc:2406] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 01:50:05.411310 20853 tablet.cc:2406] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:50:05.427587 21052 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:50:05.428049 21052 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:05.430353 21052 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7d05e51ca25403e993415676dcaaf6e (127.20.1.190:46541), baaadc5c42084ad8b930708422b8fda6 (127.20.1.188:44117)
I20260620 01:50:05.432817 20874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7675b441a14937960e9dbe0cae4ea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d05e51ca25403e993415676dcaaf6e" is_pre_election: true
I20260620 01:50:05.433652 20874 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f7675b441a14937960e9dbe0cae4ea8 in term 0.
I20260620 01:50:05.434613 20920 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7675b441a14937960e9dbe0cae4ea8, d7d05e51ca25403e993415676dcaaf6e; no voters:
I20260620 01:50:05.436035 21016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7675b441a14937960e9dbe0cae4ea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baaadc5c42084ad8b930708422b8fda6" is_pre_election: true
I20260620 01:50:05.436672 21016 raft_consensus.cc:2468] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f7675b441a14937960e9dbe0cae4ea8 in term 0.
I20260620 01:50:05.438124 21052 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:50:05.438342 21052 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:50:05.438542 21052 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:50:05.451494 21052 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:05.455825 20874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7675b441a14937960e9dbe0cae4ea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d05e51ca25403e993415676dcaaf6e"
I20260620 01:50:05.456132 20874 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:50:05.460443 20874 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7675b441a14937960e9dbe0cae4ea8 in term 1.
I20260620 01:50:05.461675 20920 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7675b441a14937960e9dbe0cae4ea8, d7d05e51ca25403e993415676dcaaf6e; no voters:
I20260620 01:50:05.468986 21016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7675b441a14937960e9dbe0cae4ea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baaadc5c42084ad8b930708422b8fda6"
I20260620 01:50:05.469331 21016 raft_consensus.cc:3060] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:50:05.473438 21016 raft_consensus.cc:2468] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7675b441a14937960e9dbe0cae4ea8 in term 1.
I20260620 01:50:05.475107 21052 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [CANDIDATE]: Term 1 election: Requested vote from peers d7d05e51ca25403e993415676dcaaf6e (127.20.1.190:46541), baaadc5c42084ad8b930708422b8fda6 (127.20.1.188:44117)
I20260620 01:50:05.475531 21052 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:50:05.475723 21052 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 1 LEADER]: Becoming Leader. State: Replica: 4f7675b441a14937960e9dbe0cae4ea8, State: Running, Role: LEADER
I20260620 01:50:05.476393 21052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:05.504809 21091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7675b441a14937960e9dbe0cae4ea8. Latest consensus state: current_term: 1 leader_uuid: "4f7675b441a14937960e9dbe0cae4ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:05.505638 21091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:05.517567 21093 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:50:05.521140 21093 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:50:05.553333 21016 raft_consensus.cc:1275] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 1 FOLLOWER]: Refusing update from remote peer 4f7675b441a14937960e9dbe0cae4ea8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:50:05.554611 21091 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:50:05.565842 20874 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 1 FOLLOWER]: Refusing update from remote peer 4f7675b441a14937960e9dbe0cae4ea8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:50:05.567057 21091 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:50:05.605564 21055 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7675b441a14937960e9dbe0cae4ea8. Latest consensus state: current_term: 1 leader_uuid: "4f7675b441a14937960e9dbe0cae4ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:05.606000 21055 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:05.612440 21094 mvcc.cc:204] Tried to move back new op lower bound from 7298745161929019392 to 7298745161693982720. Current Snapshot: MvccSnapshot[applied={T|T < 7298745161929019392}]
I20260620 01:50:05.614995 21096 mvcc.cc:204] Tried to move back new op lower bound from 7298745161929019392 to 7298745161693982720. Current Snapshot: MvccSnapshot[applied={T|T < 7298745161929019392}]
I20260620 01:50:05.625065 21091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f7675b441a14937960e9dbe0cae4ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:05.625449 21091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:05.625694 21055 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f7675b441a14937960e9dbe0cae4ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:05.625933 21055 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:05.630625 21053 sys_catalog.cc:455] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7675b441a14937960e9dbe0cae4ea8. Latest consensus state: current_term: 1 leader_uuid: "4f7675b441a14937960e9dbe0cae4ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:05.631026 21053 sys_catalog.cc:458] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:05.631466 21093 catalog_manager.cc:1382] Generated new cluster ID: e823db54e5164b59a86f95df26cb7cc5
I20260620 01:50:05.631623 21093 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:50:05.633926 21052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f7675b441a14937960e9dbe0cae4ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:05.634292 21052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:05.645264 21053 sys_catalog.cc:455] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f7675b441a14937960e9dbe0cae4ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:05.645602 21053 sys_catalog.cc:458] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:05.702105 21093 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:50:05.704250 21093 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:50:05.753513 21093 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8: Generated new TSK 0
I20260620 01:50:05.757937 21093 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:50:05.774562 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--time_source=system_unsync with env {}
I20260620 01:50:06.296607 21086 catalog_manager.cc:1294] Loaded cluster ID: e823db54e5164b59a86f95df26cb7cc5
I20260620 01:50:06.296783 21086 catalog_manager.cc:1587] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6: loading cluster ID for follower catalog manager: success
I20260620 01:50:06.301095 21086 catalog_manager.cc:1609] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6: acquiring CA information for follower catalog manager: success
I20260620 01:50:06.303956 21086 catalog_manager.cc:1637] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:50:06.326253 21089 catalog_manager.cc:1294] Loaded cluster ID: e823db54e5164b59a86f95df26cb7cc5
I20260620 01:50:06.326449 21089 catalog_manager.cc:1587] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: loading cluster ID for follower catalog manager: success
I20260620 01:50:06.331421 21089 catalog_manager.cc:1609] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: acquiring CA information for follower catalog manager: success
I20260620 01:50:06.336519 21089 catalog_manager.cc:1637] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 01:50:06.369684 21102 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:06.370121 21102 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:06.370256 21102 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:06.381240 21102 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:06.381471 21102 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:06.395238 21102 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:06.397269 21102 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:06.399564 21102 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:06.413375 21107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:06.414312 21108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:06.414702 21110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:06.415704 21102 server_base.cc:1061] running on GCE node
I20260620 01:50:06.417078 21102 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:06.417367 21102 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:06.417469 21102 hybrid_clock.cc:648] HybridClock initialized: now 1781920206417427 us; error 0 us; skew 500 ppm
I20260620 01:50:06.417878 21102 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:06.421164 21102 webserver.cc:533] Webserver started at http://127.20.1.129:46769/ using document root <none> and password file <none>
I20260620 01:50:06.422166 21102 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:06.422322 21102 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:06.422731 21102 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:06.425836 21102 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "b377b8e640814c82a541b0847e881666"
format_stamp: "Formatted at 2026-06-20 01:50:06 on dist-test-slave-wg14"
I20260620 01:50:06.426893 21102 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "b377b8e640814c82a541b0847e881666"
format_stamp: "Formatted at 2026-06-20 01:50:06 on dist-test-slave-wg14"
I20260620 01:50:06.434369 21102 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.005s
I20260620 01:50:06.441404 21115 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:06.444008 21102 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.002s
I20260620 01:50:06.444254 21102 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "b377b8e640814c82a541b0847e881666"
format_stamp: "Formatted at 2026-06-20 01:50:06 on dist-test-slave-wg14"
I20260620 01:50:06.444525 21102 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:06.475304 21102 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:06.476583 21102 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:06.476909 21102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:06.478492 21102 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:06.481019 21102 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:06.481215 21102 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:06.481359 21102 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:06.481449 21102 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:06.532976 21102 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:39995
I20260620 01:50:06.533025 21227 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:39995 every 8 connection(s)
I20260620 01:50:06.535462 21102 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:06.539475 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 21102
I20260620 01:50:06.539814 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:50:06.548820 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--time_source=system_unsync with env {}
I20260620 01:50:06.581020 21229 heartbeater.cc:344] Connected to a master server at 127.20.1.190:46541
I20260620 01:50:06.581578 21229 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:06.582886 21229 heartbeater.cc:507] Master 127.20.1.190:46541 requested a full tablet report, sending...
I20260620 01:50:06.583851 21230 heartbeater.cc:344] Connected to a master server at 127.20.1.189:44845
I20260620 01:50:06.584036 21230 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:06.584741 21230 heartbeater.cc:507] Master 127.20.1.189:44845 requested a full tablet report, sending...
I20260620 01:50:06.586191 20863 ts_manager.cc:194] Registered new tserver with Master: b377b8e640814c82a541b0847e881666 (127.20.1.129:39995)
I20260620 01:50:06.586896 21228 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44117
I20260620 01:50:06.587131 21228 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:06.587054 20933 ts_manager.cc:194] Registered new tserver with Master: b377b8e640814c82a541b0847e881666 (127.20.1.129:39995)
I20260620 01:50:06.587762 21228 heartbeater.cc:507] Master 127.20.1.188:44117 requested a full tablet report, sending...
I20260620 01:50:06.590256 20933 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:53157
I20260620 01:50:06.590447 21006 ts_manager.cc:194] Registered new tserver with Master: b377b8e640814c82a541b0847e881666 (127.20.1.129:39995)
W20260620 01:50:07.038259 21238 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:07.038756 21238 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:07.038856 21238 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:07.051043 21238 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:07.051415 21238 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:50:07.065613 21238 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:07.067600 21238 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:07.071973 21238 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:07.086215 21244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:07.087015 21243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:07.088459 21246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:07.090261 21238 server_base.cc:1061] running on GCE node
I20260620 01:50:07.091044 21238 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:07.091338 21238 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:07.091440 21238 hybrid_clock.cc:648] HybridClock initialized: now 1781920207091417 us; error 0 us; skew 500 ppm
I20260620 01:50:07.091773 21238 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:07.095873 21238 webserver.cc:533] Webserver started at http://127.20.1.130:37891/ using document root <none> and password file <none>
I20260620 01:50:07.097018 21238 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:07.097184 21238 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:07.097683 21238 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:07.102346 21238 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/data/instance:
uuid: "8f13a66dd2e143e6b82c4a96325346c7"
format_stamp: "Formatted at 2026-06-20 01:50:07 on dist-test-slave-wg14"
I20260620 01:50:07.104195 21238 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance:
uuid: "8f13a66dd2e143e6b82c4a96325346c7"
format_stamp: "Formatted at 2026-06-20 01:50:07 on dist-test-slave-wg14"
I20260620 01:50:07.116070 21238 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.011s sys 0.000s
I20260620 01:50:07.121726 21251 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:07.124642 21238 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:50:07.125059 21238 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "8f13a66dd2e143e6b82c4a96325346c7"
format_stamp: "Formatted at 2026-06-20 01:50:07 on dist-test-slave-wg14"
I20260620 01:50:07.125590 21238 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:07.152441 21238 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:07.155110 21238 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:07.156206 21238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:07.158667 21238 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:07.161306 21238 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:07.161484 21238 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:07.161594 21238 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:07.161659 21238 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:07.223275 21238 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:39081
I20260620 01:50:07.223305 21363 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:39081 every 8 connection(s)
I20260620 01:50:07.225430 21238 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:50:07.235247 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 21238
I20260620 01:50:07.235509 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
I20260620 01:50:07.241681 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--time_source=system_unsync with env {}
I20260620 01:50:07.280594 21366 heartbeater.cc:344] Connected to a master server at 127.20.1.189:44845
I20260620 01:50:07.280642 21365 heartbeater.cc:344] Connected to a master server at 127.20.1.190:46541
I20260620 01:50:07.280694 21364 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44117
I20260620 01:50:07.281214 21364 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:07.281216 21366 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:07.281618 21365 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:07.282866 21366 heartbeater.cc:507] Master 127.20.1.189:44845 requested a full tablet report, sending...
I20260620 01:50:07.282915 21364 heartbeater.cc:507] Master 127.20.1.188:44117 requested a full tablet report, sending...
I20260620 01:50:07.282939 21365 heartbeater.cc:507] Master 127.20.1.190:46541 requested a full tablet report, sending...
I20260620 01:50:07.285943 20933 ts_manager.cc:194] Registered new tserver with Master: 8f13a66dd2e143e6b82c4a96325346c7 (127.20.1.130:39081)
I20260620 01:50:07.286500 21006 ts_manager.cc:194] Registered new tserver with Master: 8f13a66dd2e143e6b82c4a96325346c7 (127.20.1.130:39081)
I20260620 01:50:07.287817 20933 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.130:49979
I20260620 01:50:07.288190 20863 ts_manager.cc:194] Registered new tserver with Master: 8f13a66dd2e143e6b82c4a96325346c7 (127.20.1.130:39081)
I20260620 01:50:07.595572 21230 heartbeater.cc:499] Master 127.20.1.189:44845 was elected leader, sending a full tablet report...
W20260620 01:50:07.761425 21374 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:07.761830 21374 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:07.761950 21374 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:07.772517 21374 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:07.772815 21374 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:50:07.788908 21374 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:07.791567 21374 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:07.795073 21374 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:07.812606 21379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:07.812636 21382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:07.812731 21380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:07.814888 21374 server_base.cc:1061] running on GCE node
I20260620 01:50:07.816027 21374 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:07.816264 21374 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:07.816370 21374 hybrid_clock.cc:648] HybridClock initialized: now 1781920207816343 us; error 0 us; skew 500 ppm
I20260620 01:50:07.816759 21374 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:07.820489 21374 webserver.cc:533] Webserver started at http://127.20.1.131:35383/ using document root <none> and password file <none>
I20260620 01:50:07.821578 21374 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:07.821710 21374 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:07.822103 21374 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:07.825410 21374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/data/instance:
uuid: "d2c812fa213e4678a38a2db521958dc9"
format_stamp: "Formatted at 2026-06-20 01:50:07 on dist-test-slave-wg14"
I20260620 01:50:07.826484 21374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance:
uuid: "d2c812fa213e4678a38a2db521958dc9"
format_stamp: "Formatted at 2026-06-20 01:50:07 on dist-test-slave-wg14"
I20260620 01:50:07.834424 21374 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260620 01:50:07.840902 21387 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:07.843017 21374 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260620 01:50:07.843330 21374 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "d2c812fa213e4678a38a2db521958dc9"
format_stamp: "Formatted at 2026-06-20 01:50:07 on dist-test-slave-wg14"
I20260620 01:50:07.843613 21374 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:07.900961 21374 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:07.902300 21374 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:07.902673 21374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:07.904156 21374 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:07.906483 21374 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:07.906644 21374 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:07.906775 21374 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:07.906859 21374 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:07.958717 21374 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:37529
I20260620 01:50:07.958729 21499 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:37529 every 8 connection(s)
I20260620 01:50:07.960894 21374 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:50:07.963049 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 21374
I20260620 01:50:07.963583 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
I20260620 01:50:08.017696 21500 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44117
I20260620 01:50:08.017976 21501 heartbeater.cc:344] Connected to a master server at 127.20.1.190:46541
I20260620 01:50:08.018256 21502 heartbeater.cc:344] Connected to a master server at 127.20.1.189:44845
I20260620 01:50:08.018738 21501 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:08.018874 21500 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:08.018734 21502 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:08.021814 21500 heartbeater.cc:507] Master 127.20.1.188:44117 requested a full tablet report, sending...
I20260620 01:50:08.022042 21502 heartbeater.cc:507] Master 127.20.1.189:44845 requested a full tablet report, sending...
I20260620 01:50:08.021840 21501 heartbeater.cc:507] Master 127.20.1.190:46541 requested a full tablet report, sending...
I20260620 01:50:08.024916 20863 ts_manager.cc:194] Registered new tserver with Master: d2c812fa213e4678a38a2db521958dc9 (127.20.1.131:37529)
I20260620 01:50:08.025032 21005 ts_manager.cc:194] Registered new tserver with Master: d2c812fa213e4678a38a2db521958dc9 (127.20.1.131:37529)
I20260620 01:50:08.025936 20933 ts_manager.cc:194] Registered new tserver with Master: d2c812fa213e4678a38a2db521958dc9 (127.20.1.131:37529)
I20260620 01:50:08.027304 20933 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.131:34491
I20260620 01:50:08.031569 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:50:08.046389 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:50:08.046741 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 20834
I20260620 01:50:08.048323 20894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:08.185869 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 20834
I20260620 01:50:08.210924 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:46541
--webserver_interface=127.20.1.190
--webserver_port=44931
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117 with env {}
I20260620 01:50:08.292829 21366 heartbeater.cc:499] Master 127.20.1.189:44845 was elected leader, sending a full tablet report...
W20260620 01:50:08.293094 21365 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:46541 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:46541: connect: Connection refused (error 111)
W20260620 01:50:08.595301 21229 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:46541 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:46541: connect: Connection refused (error 111)
W20260620 01:50:08.665951 21511 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:08.666399 21511 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:08.666523 21511 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:08.678385 21511 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:08.678565 21511 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:08.678663 21511 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:08.678732 21511 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:08.691751 21511 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:46541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=44931
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:08.694294 21511 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:08.698807 21511 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:08.711948 21519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:08.712356 21518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:08.712881 21521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:08.713490 21511 server_base.cc:1061] running on GCE node
I20260620 01:50:08.714437 21511 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:08.714624 21511 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:08.714705 21511 hybrid_clock.cc:648] HybridClock initialized: now 1781920208714682 us; error 0 us; skew 500 ppm
I20260620 01:50:08.715085 21511 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:08.719478 21511 webserver.cc:533] Webserver started at http://127.20.1.190:44931/ using document root <none> and password file <none>
I20260620 01:50:08.720438 21511 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:08.720575 21511 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:08.729995 21511 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.000s sys 0.008s
I20260620 01:50:08.734921 21528 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20260620 01:50:08.735018 20920 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 -> Peer d7d05e51ca25403e993415676dcaaf6e (127.20.1.190:46541): Couldn't send request to peer d7d05e51ca25403e993415676dcaaf6e. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:46541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 01:50:08.738276 21511 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260620 01:50:08.738682 21511 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "d7d05e51ca25403e993415676dcaaf6e"
format_stamp: "Formatted at 2026-06-20 01:50:03 on dist-test-slave-wg14"
I20260620 01:50:08.740312 21511 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:08.765755 21511 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:08.766850 21511 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:08.767201 21511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:08.798110 21511 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:46541
I20260620 01:50:08.798134 21579 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:46541 every 8 connection(s)
I20260620 01:50:08.800479 21511 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:50:08.801209 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 21511
I20260620 01:50:08.801563 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 21102
I20260620 01:50:08.810963 21580 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:50:08.814107 21223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:08.816627 21580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: Bootstrap starting.
I20260620 01:50:08.841467 21580 log.cc:826] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:08.855682 21580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:50:08.856695 21580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: Bootstrap complete.
I20260620 01:50:08.865913 21580 raft_consensus.cc:359] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:08.867056 21580 raft_consensus.cc:740] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d05e51ca25403e993415676dcaaf6e, State: Initialized, Role: FOLLOWER
I20260620 01:50:08.868631 21580 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:08.870100 21586 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:08.870667 21586 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:08.872037 21580 sys_catalog.cc:565] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:08.902246 21597 catalog_manager.cc:1294] Loaded cluster ID: e823db54e5164b59a86f95df26cb7cc5
I20260620 01:50:08.902542 21597 catalog_manager.cc:1587] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: loading cluster ID for follower catalog manager: success
I20260620 01:50:08.911666 21597 catalog_manager.cc:1609] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: acquiring CA information for follower catalog manager: success
I20260620 01:50:08.915629 21597 catalog_manager.cc:1637] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:50:08.949774 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 21102
I20260620 01:50:08.975661 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:39995
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--webserver_port=46769
--webserver_interface=127.20.1.129
--time_source=system_unsync with env {}
I20260620 01:50:09.030798 21502 heartbeater.cc:499] Master 127.20.1.189:44845 was elected leader, sending a full tablet report...
I20260620 01:50:09.050093 21545 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d2c812fa213e4678a38a2db521958dc9" instance_seqno: 1781920207945294) as {username='slave'} at 127.20.1.131:59475; Asking this server to re-register.
I20260620 01:50:09.051600 21501 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:09.054478 21545 ts_manager.cc:194] Registered new tserver with Master: d2c812fa213e4678a38a2db521958dc9 (127.20.1.131:37529)
I20260620 01:50:09.114253 21091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f7675b441a14937960e9dbe0cae4ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:09.114574 21091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:09.129732 21586 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7675b441a14937960e9dbe0cae4ea8. Latest consensus state: current_term: 1 leader_uuid: "4f7675b441a14937960e9dbe0cae4ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:09.130079 21586 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:09.308147 21365 heartbeater.cc:344] Connected to a master server at 127.20.1.190:46541
I20260620 01:50:09.310534 21545 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8f13a66dd2e143e6b82c4a96325346c7" instance_seqno: 1781920207209714) as {username='slave'} at 127.20.1.130:60931; Asking this server to re-register.
I20260620 01:50:09.311511 21365 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:09.313274 21545 ts_manager.cc:194] Registered new tserver with Master: 8f13a66dd2e143e6b82c4a96325346c7 (127.20.1.130:39081)
W20260620 01:50:09.543006 21603 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:09.543633 21603 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:09.544694 21603 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:09.557713 21603 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:09.558036 21603 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:09.576350 21603 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:39995
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=46769
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:09.578536 21603 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:09.589797 21606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f7675b441a14937960e9dbe0cae4ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:09.590150 21606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:09.591789 21603 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:09.615638 21616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:09.615664 21614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:09.615584 21613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:09.616516 21603 server_base.cc:1061] running on GCE node
I20260620 01:50:09.617282 21603 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:09.617480 21603 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:09.617565 21603 hybrid_clock.cc:648] HybridClock initialized: now 1781920209617543 us; error 0 us; skew 500 ppm
I20260620 01:50:09.617924 21603 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:09.626885 21603 webserver.cc:533] Webserver started at http://127.20.1.129:46769/ using document root <none> and password file <none>
I20260620 01:50:09.629731 21603 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:09.629927 21603 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:09.637284 21603 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 01:50:09.644906 21621 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:09.647487 21603 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.006s sys 0.000s
I20260620 01:50:09.647739 21603 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "b377b8e640814c82a541b0847e881666"
format_stamp: "Formatted at 2026-06-20 01:50:06 on dist-test-slave-wg14"
I20260620 01:50:09.648758 21603 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:09.689591 21603 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:09.690761 21603 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:09.691089 21603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:09.692687 21603 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:09.695294 21603 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:09.695439 21603 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:09.695559 21603 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:09.695614 21603 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:09.757686 21603 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:39995
I20260620 01:50:09.757699 21733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:39995 every 8 connection(s)
I20260620 01:50:09.760147 21603 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:09.768224 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 21603
I20260620 01:50:09.768683 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 21603
I20260620 01:50:09.792695 21734 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44117
I20260620 01:50:09.792714 21736 heartbeater.cc:344] Connected to a master server at 127.20.1.189:44845
I20260620 01:50:09.793799 21734 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:09.793799 21736 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:09.796168 21734 heartbeater.cc:507] Master 127.20.1.188:44117 requested a full tablet report, sending...
I20260620 01:50:09.798177 21736 heartbeater.cc:507] Master 127.20.1.189:44845 requested a full tablet report, sending...
I20260620 01:50:09.799012 21729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:09.799368 21006 ts_manager.cc:194] Re-registered known tserver with Master: b377b8e640814c82a541b0847e881666 (127.20.1.129:39995)
I20260620 01:50:09.800043 20933 ts_manager.cc:194] Re-registered known tserver with Master: b377b8e640814c82a541b0847e881666 (127.20.1.129:39995)
I20260620 01:50:09.801333 20933 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:35561
I20260620 01:50:09.908892 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 21603
I20260620 01:50:09.925323 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:39995
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--webserver_port=46769
--webserver_interface=127.20.1.129
--time_source=system_unsync
--fault_before_start=1.0 with env {}
W20260620 01:50:10.532281 21746 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:10.532661 21746 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:10.532756 21746 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:50:10.532825 21746 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:10.546900 21746 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:10.547478 21746 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:10.567773 21746 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:39995
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=46769
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:10.570830 21746 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:10.573534 21746 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:10.595274 21753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:10.595463 21752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:10.596145 21755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:10.597293 21746 server_base.cc:1061] running on GCE node
I20260620 01:50:10.598483 21746 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:10.609069 21746 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:10.609280 21746 hybrid_clock.cc:648] HybridClock initialized: now 1781920210609226 us; error 0 us; skew 500 ppm
I20260620 01:50:10.610023 21746 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:10.614347 21746 webserver.cc:533] Webserver started at http://127.20.1.129:46769/ using document root <none> and password file <none>
I20260620 01:50:10.615739 21746 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:10.616153 21746 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:10.634990 21746 fs_manager.cc:714] Time spent opening directory manager: real 0.013s user 0.011s sys 0.000s
I20260620 01:50:10.643064 21760 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:10.645543 21746 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.003s sys 0.004s
I20260620 01:50:10.645833 21746 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "b377b8e640814c82a541b0847e881666"
format_stamp: "Formatted at 2026-06-20 01:50:06 on dist-test-slave-wg14"
I20260620 01:50:10.646871 21746 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:10.669092 21746 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:10.670498 21746 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:10.670795 21746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:10.672046 21746 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:10.674825 21746 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:10.674980 21746 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:10.675101 21746 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:10.675201 21746 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.002s
W20260620 01:50:10.675719 21746 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:50:10.675925 21746 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:50:10.676002 21746 fault_injection.cc:59] injecting fault for kudu (pid 21746): FLAGS_fault_before_start (process will exit)
I20260620 01:50:10.699611 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:39995
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--webserver_port=46769
--webserver_interface=127.20.1.129
--time_source=system_unsync with env {}
W20260620 01:50:11.166811 21768 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:11.167294 21768 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:11.167448 21768 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:11.178014 21768 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:11.178248 21768 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:11.190164 21768 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:39995
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=46769
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:11.193426 21768 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:11.196259 21768 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:11.214196 21773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:11.216053 21776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:11.216176 21774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:11.218464 21768 server_base.cc:1061] running on GCE node
I20260620 01:50:11.219470 21768 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:11.219708 21768 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:11.219810 21768 hybrid_clock.cc:648] HybridClock initialized: now 1781920211219783 us; error 0 us; skew 500 ppm
I20260620 01:50:11.220155 21768 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:11.223699 21768 webserver.cc:533] Webserver started at http://127.20.1.129:46769/ using document root <none> and password file <none>
I20260620 01:50:11.225131 21768 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:11.225369 21768 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:11.236159 21768 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.007s sys 0.000s
I20260620 01:50:11.243242 21781 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:11.246455 21768 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260620 01:50:11.246739 21768 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "b377b8e640814c82a541b0847e881666"
format_stamp: "Formatted at 2026-06-20 01:50:06 on dist-test-slave-wg14"
I20260620 01:50:11.248265 21768 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:11.278328 21768 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:11.279520 21768 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:11.279846 21768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:11.281960 21768 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:11.284554 21768 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:11.284724 21768 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:11.284837 21768 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:11.284886 21768 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:11.345357 21768 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:39995
I20260620 01:50:11.345427 21893 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:39995 every 8 connection(s)
I20260620 01:50:11.347486 21768 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:11.350924 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 21768
I20260620 01:50:11.351464 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 21768
I20260620 01:50:11.388368 21895 heartbeater.cc:344] Connected to a master server at 127.20.1.190:46541
I20260620 01:50:11.388876 21895 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:11.391119 21895 heartbeater.cc:507] Master 127.20.1.190:46541 requested a full tablet report, sending...
I20260620 01:50:11.392055 21896 heartbeater.cc:344] Connected to a master server at 127.20.1.189:44845
I20260620 01:50:11.392352 21896 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:11.393158 21896 heartbeater.cc:507] Master 127.20.1.189:44845 requested a full tablet report, sending...
I20260620 01:50:11.394307 21889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:11.395179 21545 ts_manager.cc:194] Registered new tserver with Master: b377b8e640814c82a541b0847e881666 (127.20.1.129:39995)
I20260620 01:50:11.395365 20934 ts_manager.cc:194] Re-registered known tserver with Master: b377b8e640814c82a541b0847e881666 (127.20.1.129:39995)
I20260620 01:50:11.396926 20934 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:57647
I20260620 01:50:11.508186 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 21768
W20260620 01:50:11.524435 20991 connection.cc:573] server connection from 127.20.1.129:60121 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:50:11.525373 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 21238
I20260620 01:50:11.540563 21359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:11.643837 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 21238
I20260620 01:50:11.662139 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 21374
I20260620 01:50:11.676352 21495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:11.783025 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 21374
I20260620 01:50:11.801039 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 21511
I20260620 01:50:11.813010 21575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:11.955056 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 21511
W20260620 01:50:11.978034 20920 connection.cc:573] client connection to 127.20.1.190:46541 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 01:50:11.978256 20920 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 5 similar messages]
I20260620 01:50:11.978302 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 20901
I20260620 01:50:11.979936 20964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:12.153239 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 20901
W20260620 01:50:12.153231 20920 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 -> Peer d7d05e51ca25403e993415676dcaaf6e (127.20.1.190:46541): Couldn't send request to peer d7d05e51ca25403e993415676dcaaf6e. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:46541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 01:50:12.173281 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 20971
I20260620 01:50:12.175246 21036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:12.326189 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 20971
I20260620 01:50:12.351704 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:46541
--webserver_interface=127.20.1.190
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117 with env {}
W20260620 01:50:12.792155 21917 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:12.793056 21917 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:12.793233 21917 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:12.804477 21917 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:12.804621 21917 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:12.804704 21917 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:12.804771 21917 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:12.818156 21917 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:46541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:12.821045 21917 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:12.824201 21917 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:12.836637 21923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:12.836637 21922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:12.837285 21925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:12.838959 21917 server_base.cc:1061] running on GCE node
I20260620 01:50:12.840368 21917 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:12.840605 21917 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:12.840720 21917 hybrid_clock.cc:648] HybridClock initialized: now 1781920212840692 us; error 0 us; skew 500 ppm
I20260620 01:50:12.841179 21917 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:12.845108 21917 webserver.cc:533] Webserver started at http://127.20.1.190:41461/ using document root <none> and password file <none>
I20260620 01:50:12.846094 21917 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:12.846225 21917 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:12.854863 21917 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.004s
I20260620 01:50:12.861279 21930 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:12.863777 21917 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260620 01:50:12.864037 21917 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "d7d05e51ca25403e993415676dcaaf6e"
format_stamp: "Formatted at 2026-06-20 01:50:03 on dist-test-slave-wg14"
I20260620 01:50:12.865105 21917 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:12.889719 21917 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:12.890905 21917 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:12.891268 21917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:12.917438 21917 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:46541
I20260620 01:50:12.917440 21981 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:46541 every 8 connection(s)
I20260620 01:50:12.920063 21917 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:50:12.921195 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 21917
I20260620 01:50:12.921639 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:50:12.923389 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:44845
--webserver_interface=127.20.1.189
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117 with env {}
I20260620 01:50:12.929522 21982 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:50:12.935662 21982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: Bootstrap starting.
I20260620 01:50:12.956462 21982 log.cc:826] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:12.968423 21982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:50:12.969130 21982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: Bootstrap complete.
I20260620 01:50:12.978637 21982 raft_consensus.cc:359] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:12.979658 21982 raft_consensus.cc:740] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d05e51ca25403e993415676dcaaf6e, State: Initialized, Role: FOLLOWER
I20260620 01:50:12.980933 21982 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:12.982146 21986 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:12.982570 21986 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:12.983596 21982 sys_catalog.cc:565] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:12.998205 21997 catalog_manager.cc:1294] Loaded cluster ID: e823db54e5164b59a86f95df26cb7cc5
I20260620 01:50:12.998545 21997 catalog_manager.cc:1587] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: loading cluster ID for follower catalog manager: success
I20260620 01:50:13.005214 21997 catalog_manager.cc:1609] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: acquiring CA information for follower catalog manager: success
I20260620 01:50:13.012851 21997 catalog_manager.cc:1637] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:50:13.028458 21917 master_runner.cc:187] Error getting master registration for 127.20.1.189:44845: Network error: Client connection negotiation failed: client connection to 127.20.1.189:44845: connect: Connection refused (error 111),
I20260620 01:50:13.033149 21917 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:50:13.465426 21984 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:13.465906 21984 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:13.466041 21984 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:13.477676 21984 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:13.477880 21984 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:13.477983 21984 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:13.478061 21984 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:13.491731 21984 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:44845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:13.493793 21984 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:13.497579 21984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:13.508896 22005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:13.512966 22006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:13.513643 22008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:13.515272 21984 server_base.cc:1061] running on GCE node
I20260620 01:50:13.516584 21984 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:13.516912 21984 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:13.517024 21984 hybrid_clock.cc:648] HybridClock initialized: now 1781920213516995 us; error 0 us; skew 500 ppm
I20260620 01:50:13.517584 21984 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:13.521098 21984 webserver.cc:533] Webserver started at http://127.20.1.189:35525/ using document root <none> and password file <none>
I20260620 01:50:13.522140 21984 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:13.522284 21984 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:13.529716 21984 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:50:13.534221 22013 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:13.536446 21984 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 01:50:13.536747 21984 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "4f7675b441a14937960e9dbe0cae4ea8"
format_stamp: "Formatted at 2026-06-20 01:50:04 on dist-test-slave-wg14"
I20260620 01:50:13.538071 21984 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:13.562036 21984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:13.563365 21984 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:13.563724 21984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:13.591564 21984 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:44845
I20260620 01:50:13.591657 22064 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:44845 every 8 connection(s)
I20260620 01:50:13.594075 21984 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:50:13.604027 22065 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:50:13.604333 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 21984
I20260620 01:50:13.604600 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:50:13.606475 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:44117
--webserver_interface=127.20.1.188
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117 with env {}
I20260620 01:50:13.611255 22065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8: Bootstrap starting.
I20260620 01:50:13.637540 22065 log.cc:826] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:13.650012 22065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:50:13.650722 22065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8: Bootstrap complete.
I20260620 01:50:13.658586 22065 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:13.659531 22065 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7675b441a14937960e9dbe0cae4ea8, State: Initialized, Role: FOLLOWER
I20260620 01:50:13.660848 22065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:13.663369 22069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:13.663919 22069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:13.665315 22065 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:13.687811 22080 catalog_manager.cc:1294] Loaded cluster ID: e823db54e5164b59a86f95df26cb7cc5
I20260620 01:50:13.688941 22080 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8: loading cluster ID for follower catalog manager: success
I20260620 01:50:13.694465 22080 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8: acquiring CA information for follower catalog manager: success
I20260620 01:50:13.698378 22080 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:50:13.738153 21984 master_runner.cc:187] Error getting master registration for 127.20.1.188:44117: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44117: connect: Connection refused (error 111),
I20260620 01:50:13.745416 21984 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:50:14.080392 22067 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:14.080904 22067 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:14.081070 22067 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:14.094360 22067 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:14.094561 22067 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:14.094657 22067 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:14.094725 22067 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:14.106945 22067 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:44117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:14.109077 22067 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:14.111469 22067 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:14.123711 22090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:14.123746 22089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:14.127324 22092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:14.127523 22067 server_base.cc:1061] running on GCE node
I20260620 01:50:14.128907 22067 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:14.129168 22067 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:14.129277 22067 hybrid_clock.cc:648] HybridClock initialized: now 1781920214129247 us; error 0 us; skew 500 ppm
I20260620 01:50:14.129669 22067 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:14.135056 22067 webserver.cc:533] Webserver started at http://127.20.1.188:39945/ using document root <none> and password file <none>
I20260620 01:50:14.136900 22067 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:14.137267 22067 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:14.146096 22067 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:50:14.150396 22097 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:14.152593 22067 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:50:14.152868 22067 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "baaadc5c42084ad8b930708422b8fda6"
format_stamp: "Formatted at 2026-06-20 01:50:04 on dist-test-slave-wg14"
I20260620 01:50:14.153834 22067 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:14.178196 22067 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:14.179535 22067 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:14.179873 22067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:14.207311 22067 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:44117
I20260620 01:50:14.207304 22148 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:44117 every 8 connection(s)
I20260620 01:50:14.209429 22067 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:50:14.217774 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 22067
I20260620 01:50:14.218032 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:50:14.221338 22149 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:50:14.227782 22149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6: Bootstrap starting.
I20260620 01:50:14.259718 22149 log.cc:826] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:14.276546 22149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:50:14.277396 22149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6: Bootstrap complete.
I20260620 01:50:14.286192 22149 raft_consensus.cc:359] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:14.287180 22149 raft_consensus.cc:740] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: baaadc5c42084ad8b930708422b8fda6, State: Initialized, Role: FOLLOWER
I20260620 01:50:14.288249 22149 consensus_queue.cc:260] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:14.289603 22155 sys_catalog.cc:455] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:14.290195 22155 sys_catalog.cc:458] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:14.292819 22149 sys_catalog.cc:565] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:14.308614 22166 catalog_manager.cc:1294] Loaded cluster ID: e823db54e5164b59a86f95df26cb7cc5
I20260620 01:50:14.308763 22166 catalog_manager.cc:1587] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6: loading cluster ID for follower catalog manager: success
I20260620 01:50:14.313876 22166 catalog_manager.cc:1609] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6: acquiring CA information for follower catalog manager: success
I20260620 01:50:14.317744 22166 catalog_manager.cc:1637] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:50:14.349385 22067 master_runner.cc:264] No leader master found from master baaadc5c42084ad8b930708422b8fda6
I20260620 01:50:14.349642 22067 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:50:14.513332 22169 raft_consensus.cc:493] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:50:14.513634 22169 raft_consensus.cc:515] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:14.516114 22169 leader_election.cc:290] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f7675b441a14937960e9dbe0cae4ea8 (127.20.1.189:44845), baaadc5c42084ad8b930708422b8fda6 (127.20.1.188:44117)
I20260620 01:50:14.525768 22040 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7d05e51ca25403e993415676dcaaf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4f7675b441a14937960e9dbe0cae4ea8" is_pre_election: true
I20260620 01:50:14.526520 22040 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d05e51ca25403e993415676dcaaf6e in term 1.
I20260620 01:50:14.527594 21934 leader_election.cc:304] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7675b441a14937960e9dbe0cae4ea8, d7d05e51ca25403e993415676dcaaf6e; no voters:
I20260620 01:50:14.528225 22169 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:50:14.528462 22169 raft_consensus.cc:493] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:50:14.528584 22169 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:50:14.531046 22124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7d05e51ca25403e993415676dcaaf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "baaadc5c42084ad8b930708422b8fda6" is_pre_election: true
I20260620 01:50:14.532080 22124 raft_consensus.cc:2468] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d05e51ca25403e993415676dcaaf6e in term 1.
I20260620 01:50:14.534781 22169 raft_consensus.cc:515] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:14.535856 22169 leader_election.cc:290] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [CANDIDATE]: Term 2 election: Requested vote from peers 4f7675b441a14937960e9dbe0cae4ea8 (127.20.1.189:44845), baaadc5c42084ad8b930708422b8fda6 (127.20.1.188:44117)
I20260620 01:50:14.537497 22040 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7d05e51ca25403e993415676dcaaf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4f7675b441a14937960e9dbe0cae4ea8"
I20260620 01:50:14.537580 22124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7d05e51ca25403e993415676dcaaf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "baaadc5c42084ad8b930708422b8fda6"
I20260620 01:50:14.537770 22040 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:50:14.537796 22124 raft_consensus.cc:3060] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:50:14.543298 22124 raft_consensus.cc:2468] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d05e51ca25403e993415676dcaaf6e in term 2.
I20260620 01:50:14.543998 21931 leader_election.cc:304] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: baaadc5c42084ad8b930708422b8fda6, d7d05e51ca25403e993415676dcaaf6e; no voters:
I20260620 01:50:14.544030 22040 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d05e51ca25403e993415676dcaaf6e in term 2.
I20260620 01:50:14.544430 22169 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 01:50:14.545001 22169 raft_consensus.cc:697] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 2 LEADER]: Becoming Leader. State: Replica: d7d05e51ca25403e993415676dcaaf6e, State: Running, Role: LEADER
I20260620 01:50:14.545655 22169 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:14.548949 22172 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7d05e51ca25403e993415676dcaaf6e. Latest consensus state: current_term: 2 leader_uuid: "d7d05e51ca25403e993415676dcaaf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:14.550060 22172 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:14.551031 22174 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:50:14.553270 22174 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:50:14.554932 22174 catalog_manager.cc:1294] Loaded cluster ID: e823db54e5164b59a86f95df26cb7cc5
I20260620 01:50:14.555068 22174 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:50:14.556507 22174 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:50:14.558763 22174 catalog_manager.cc:6080] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: Loaded TSK: 0
I20260620 01:50:14.560876 22174 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:50:14.590364 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--time_source=system_unsync with env {}
I20260620 01:50:15.085749 22040 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 2 FOLLOWER]: Refusing update from remote peer d7d05e51ca25403e993415676dcaaf6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:50:15.087684 22176 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:50:15.093976 22177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7d05e51ca25403e993415676dcaaf6e. Latest consensus state: current_term: 2 leader_uuid: "d7d05e51ca25403e993415676dcaaf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:15.094410 22177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:15.099038 22124 raft_consensus.cc:1275] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 2 FOLLOWER]: Refusing update from remote peer d7d05e51ca25403e993415676dcaaf6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:50:15.100302 22181 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d7d05e51ca25403e993415676dcaaf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:15.100631 22181 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:15.102555 22176 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:50:15.111929 22177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d7d05e51ca25403e993415676dcaaf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:15.112553 22177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:15.122619 22191 sys_catalog.cc:455] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d7d05e51ca25403e993415676dcaaf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:15.123572 22191 sys_catalog.cc:458] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:15.129069 22187 sys_catalog.cc:455] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7d05e51ca25403e993415676dcaaf6e. Latest consensus state: current_term: 2 leader_uuid: "d7d05e51ca25403e993415676dcaaf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
W20260620 01:50:15.129530 22175 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:15.129844 22175 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:15.129927 22175 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260620 01:50:15.129861 22187 sys_catalog.cc:458] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:50:15.142797 22175 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:15.143029 22175 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:15.157226 22175 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:15.159407 22175 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:15.162093 22175 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:15.177310 22194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:15.177367 22193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:15.183347 22196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:15.186092 22175 server_base.cc:1061] running on GCE node
I20260620 01:50:15.188215 22175 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:15.188820 22175 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:15.189081 22175 hybrid_clock.cc:648] HybridClock initialized: now 1781920215189047 us; error 0 us; skew 500 ppm
I20260620 01:50:15.190038 22175 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:15.196504 22175 webserver.cc:533] Webserver started at http://127.20.1.129:43471/ using document root <none> and password file <none>
I20260620 01:50:15.198313 22175 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:15.198544 22175 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:15.211843 22175 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.007s sys 0.000s
I20260620 01:50:15.218101 22201 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:15.221836 22175 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260620 01:50:15.222510 22175 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "b377b8e640814c82a541b0847e881666"
format_stamp: "Formatted at 2026-06-20 01:50:06 on dist-test-slave-wg14"
I20260620 01:50:15.224315 22175 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:15.273757 22175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:15.275503 22175 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:15.276113 22175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:15.278026 22175 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:15.282648 22175 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:15.282825 22175 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:15.283095 22175 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:15.283401 22175 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.002s
I20260620 01:50:15.375061 22175 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:34385
I20260620 01:50:15.375093 22313 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:34385 every 8 connection(s)
I20260620 01:50:15.377578 22175 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:15.387979 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 22175
I20260620 01:50:15.388728 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:50:15.413687 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--time_source=system_unsync with env {}
I20260620 01:50:15.426182 22315 heartbeater.cc:344] Connected to a master server at 127.20.1.190:46541
I20260620 01:50:15.427560 22315 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:15.430176 22315 heartbeater.cc:507] Master 127.20.1.190:46541 requested a full tablet report, sending...
I20260620 01:50:15.431032 22316 heartbeater.cc:344] Connected to a master server at 127.20.1.189:44845
I20260620 01:50:15.431862 22316 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:15.432754 22316 heartbeater.cc:507] Master 127.20.1.189:44845 requested a full tablet report, sending...
I20260620 01:50:15.434486 21947 ts_manager.cc:194] Registered new tserver with Master: b377b8e640814c82a541b0847e881666 (127.20.1.129:34385)
I20260620 01:50:15.437207 21947 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:59461
I20260620 01:50:15.441761 22030 ts_manager.cc:194] Registered new tserver with Master: b377b8e640814c82a541b0847e881666 (127.20.1.129:34385)
I20260620 01:50:15.444263 22314 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44117
I20260620 01:50:15.444686 22314 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:15.445547 22314 heartbeater.cc:507] Master 127.20.1.188:44117 requested a full tablet report, sending...
I20260620 01:50:15.448292 22114 ts_manager.cc:194] Registered new tserver with Master: b377b8e640814c82a541b0847e881666 (127.20.1.129:34385)
I20260620 01:50:15.730434 22325 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d7d05e51ca25403e993415676dcaaf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:15.731392 22325 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: This master's current role is: LEADER
W20260620 01:50:16.011021 22324 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:16.011540 22324 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:16.011682 22324 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:16.031492 22324 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:16.031795 22324 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:50:16.051714 22324 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:16.054912 22324 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:16.058637 22324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:16.079370 22333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:16.079325 22336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:16.079319 22334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:16.080786 22324 server_base.cc:1061] running on GCE node
I20260620 01:50:16.081986 22324 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:16.082273 22324 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:16.082361 22324 hybrid_clock.cc:648] HybridClock initialized: now 1781920216082334 us; error 0 us; skew 500 ppm
I20260620 01:50:16.082772 22324 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:16.086208 22324 webserver.cc:533] Webserver started at http://127.20.1.130:34589/ using document root <none> and password file <none>
I20260620 01:50:16.087244 22324 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:16.087391 22324 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:16.100075 22324 fs_manager.cc:714] Time spent opening directory manager: real 0.009s user 0.006s sys 0.004s
I20260620 01:50:16.106957 22341 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:16.110821 22324 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.007s sys 0.000s
I20260620 01:50:16.111177 22324 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "8f13a66dd2e143e6b82c4a96325346c7"
format_stamp: "Formatted at 2026-06-20 01:50:07 on dist-test-slave-wg14"
I20260620 01:50:16.112160 22324 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:16.149401 22324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:16.151074 22324 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:16.151515 22324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:16.152791 22324 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:16.155372 22324 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:16.155920 22324 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260620 01:50:16.156323 22324 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:16.156437 22324 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:16.227969 22324 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:36371
I20260620 01:50:16.228423 22453 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:36371 every 8 connection(s)
I20260620 01:50:16.230888 22324 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:50:16.233316 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 22324
I20260620 01:50:16.233610 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
I20260620 01:50:16.253170 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--time_source=system_unsync with env {}
I20260620 01:50:16.283699 22455 heartbeater.cc:344] Connected to a master server at 127.20.1.190:46541
I20260620 01:50:16.284413 22455 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:16.285852 22455 heartbeater.cc:507] Master 127.20.1.190:46541 requested a full tablet report, sending...
I20260620 01:50:16.286850 22454 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44117
I20260620 01:50:16.287038 22454 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:16.287810 22454 heartbeater.cc:507] Master 127.20.1.188:44117 requested a full tablet report, sending...
I20260620 01:50:16.288378 21947 ts_manager.cc:194] Registered new tserver with Master: 8f13a66dd2e143e6b82c4a96325346c7 (127.20.1.130:36371)
I20260620 01:50:16.289534 22114 ts_manager.cc:194] Registered new tserver with Master: 8f13a66dd2e143e6b82c4a96325346c7 (127.20.1.130:36371)
I20260620 01:50:16.289644 21947 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.130:60035
I20260620 01:50:16.297169 22456 heartbeater.cc:344] Connected to a master server at 127.20.1.189:44845
I20260620 01:50:16.297684 22456 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:16.298611 22456 heartbeater.cc:507] Master 127.20.1.189:44845 requested a full tablet report, sending...
I20260620 01:50:16.302008 22030 ts_manager.cc:194] Registered new tserver with Master: 8f13a66dd2e143e6b82c4a96325346c7 (127.20.1.130:36371)
I20260620 01:50:16.452173 22315 heartbeater.cc:499] Master 127.20.1.190:46541 was elected leader, sending a full tablet report...
W20260620 01:50:16.814152 22464 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:16.814648 22464 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:16.814750 22464 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:16.826555 22464 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:16.826843 22464 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:50:16.839496 22464 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:16.841284 22464 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:16.843528 22464 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:16.857739 22469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:16.857708 22470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:16.857807 22472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:16.860179 22464 server_base.cc:1061] running on GCE node
I20260620 01:50:16.861383 22464 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:16.861635 22464 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:16.861892 22464 hybrid_clock.cc:648] HybridClock initialized: now 1781920216861732 us; error 0 us; skew 500 ppm
I20260620 01:50:16.862479 22464 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:16.865689 22464 webserver.cc:533] Webserver started at http://127.20.1.131:33373/ using document root <none> and password file <none>
I20260620 01:50:16.866791 22464 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:16.866935 22464 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:16.875838 22464 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:50:16.880473 22477 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:16.882670 22464 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:50:16.882931 22464 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "d2c812fa213e4678a38a2db521958dc9"
format_stamp: "Formatted at 2026-06-20 01:50:07 on dist-test-slave-wg14"
I20260620 01:50:16.884361 22464 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:16.906891 22464 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:16.908900 22464 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:16.909374 22464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:16.911152 22464 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:16.913847 22464 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:16.913983 22464 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:16.914137 22464 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:16.914208 22464 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:16.961825 22464 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:39585
I20260620 01:50:16.961742 22589 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:39585 every 8 connection(s)
I20260620 01:50:16.965077 22464 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:50:16.966590 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 22464
I20260620 01:50:16.966933 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
I20260620 01:50:16.994850 22591 heartbeater.cc:344] Connected to a master server at 127.20.1.190:46541
I20260620 01:50:16.995246 22590 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44117
I20260620 01:50:16.995452 22590 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:16.995450 22591 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:16.995563 22592 heartbeater.cc:344] Connected to a master server at 127.20.1.189:44845
I20260620 01:50:16.995738 22592 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:16.997097 22592 heartbeater.cc:507] Master 127.20.1.189:44845 requested a full tablet report, sending...
I20260620 01:50:16.997076 22591 heartbeater.cc:507] Master 127.20.1.190:46541 requested a full tablet report, sending...
I20260620 01:50:16.997083 22590 heartbeater.cc:507] Master 127.20.1.188:44117 requested a full tablet report, sending...
I20260620 01:50:16.999626 21947 ts_manager.cc:194] Registered new tserver with Master: d2c812fa213e4678a38a2db521958dc9 (127.20.1.131:39585)
I20260620 01:50:16.999788 22114 ts_manager.cc:194] Registered new tserver with Master: d2c812fa213e4678a38a2db521958dc9 (127.20.1.131:39585)
I20260620 01:50:16.999779 22030 ts_manager.cc:194] Registered new tserver with Master: d2c812fa213e4678a38a2db521958dc9 (127.20.1.131:39585)
I20260620 01:50:17.000787 21947 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.131:56787
I20260620 01:50:17.015599 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:50:17.037467 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:50:17.037770 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 21917
I20260620 01:50:17.039397 21977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:17.165306 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 21917
W20260620 01:50:17.185276 22017 connection.cc:573] server connection from 127.0.0.1:34388 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:50:17.186146 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:46541
--webserver_interface=127.20.1.190
--webserver_port=41461
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117 with env {}
I20260620 01:50:17.296203 22455 heartbeater.cc:499] Master 127.20.1.190:46541 was elected leader, sending a full tablet report...
W20260620 01:50:17.298375 22455 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:46541 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:46541: connect: Connection refused (error 111)
W20260620 01:50:17.456512 22315 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:46541 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:46541: connect: Connection refused (error 111)
W20260620 01:50:17.656796 22601 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:17.657300 22601 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:17.657433 22601 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:17.672606 22601 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:17.672730 22601 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:17.672791 22601 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:17.672894 22601 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:17.686264 22601 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:46541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=41461
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:17.688691 22601 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:17.691339 22601 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:17.706027 22608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:17.706573 22609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:17.706615 22611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:17.708271 22601 server_base.cc:1061] running on GCE node
I20260620 01:50:17.709558 22601 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:17.709808 22601 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:17.709920 22601 hybrid_clock.cc:648] HybridClock initialized: now 1781920217709890 us; error 0 us; skew 500 ppm
I20260620 01:50:17.710361 22601 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:17.714150 22601 webserver.cc:533] Webserver started at http://127.20.1.190:41461/ using document root <none> and password file <none>
I20260620 01:50:17.715375 22601 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:17.715580 22601 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:17.724030 22601 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:50:17.728700 22616 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:17.731195 22601 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:50:17.731480 22601 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "d7d05e51ca25403e993415676dcaaf6e"
format_stamp: "Formatted at 2026-06-20 01:50:03 on dist-test-slave-wg14"
I20260620 01:50:17.732446 22601 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:17.751406 22601 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:17.752861 22601 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:17.753340 22601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:17.780725 22601 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:46541
I20260620 01:50:17.780735 22667 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:46541 every 8 connection(s)
I20260620 01:50:17.782884 22601 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:50:17.784750 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 22601
I20260620 01:50:17.785218 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 22175
I20260620 01:50:17.795223 22668 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:50:17.801986 22668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: Bootstrap starting.
I20260620 01:50:17.803728 22309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:17.822753 22668 log.cc:826] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:17.837579 22668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:50:17.838259 22668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: Bootstrap complete.
I20260620 01:50:17.846136 22668 raft_consensus.cc:359] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:17.846987 22668 raft_consensus.cc:740] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d05e51ca25403e993415676dcaaf6e, State: Initialized, Role: FOLLOWER
I20260620 01:50:17.847997 22668 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:17.849538 22674 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:17.850142 22674 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:17.850692 22668 sys_catalog.cc:565] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:17.866175 22685 catalog_manager.cc:1294] Loaded cluster ID: e823db54e5164b59a86f95df26cb7cc5
I20260620 01:50:17.866338 22685 catalog_manager.cc:1587] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: loading cluster ID for follower catalog manager: success
I20260620 01:50:17.872326 22685 catalog_manager.cc:1609] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: acquiring CA information for follower catalog manager: success
I20260620 01:50:17.876996 22685 catalog_manager.cc:1637] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:50:17.921761 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 22175
I20260620 01:50:17.933501 22601 master_runner.cc:264] No leader master found from master d7d05e51ca25403e993415676dcaaf6e
I20260620 01:50:17.933835 22601 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:50:17.947460 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:34385
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--webserver_port=43471
--webserver_interface=127.20.1.129
--time_source=system_unsync with env {}
I20260620 01:50:18.004503 22591 heartbeater.cc:499] Master 127.20.1.190:46541 was elected leader, sending a full tablet report...
I20260620 01:50:18.020900 22633 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d2c812fa213e4678a38a2db521958dc9" instance_seqno: 1781920216948698) as {username='slave'} at 127.20.1.131:34383; Asking this server to re-register.
I20260620 01:50:18.022277 22591 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:18.024791 22633 ts_manager.cc:194] Registered new tserver with Master: d2c812fa213e4678a38a2db521958dc9 (127.20.1.131:39585)
I20260620 01:50:18.255435 22693 raft_consensus.cc:493] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7d05e51ca25403e993415676dcaaf6e)
I20260620 01:50:18.255818 22693 raft_consensus.cc:515] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:18.259627 22693 leader_election.cc:290] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7d05e51ca25403e993415676dcaaf6e (127.20.1.190:46541), 4f7675b441a14937960e9dbe0cae4ea8 (127.20.1.189:44845)
I20260620 01:50:18.261487 22040 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baaadc5c42084ad8b930708422b8fda6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4f7675b441a14937960e9dbe0cae4ea8" is_pre_election: true
I20260620 01:50:18.273716 22643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baaadc5c42084ad8b930708422b8fda6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7d05e51ca25403e993415676dcaaf6e" is_pre_election: true
I20260620 01:50:18.274528 22643 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baaadc5c42084ad8b930708422b8fda6 in term 2.
I20260620 01:50:18.275499 22100 leader_election.cc:304] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: baaadc5c42084ad8b930708422b8fda6, d7d05e51ca25403e993415676dcaaf6e; no voters: 4f7675b441a14937960e9dbe0cae4ea8
I20260620 01:50:18.276119 22693 raft_consensus.cc:2804] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 01:50:18.276284 22693 raft_consensus.cc:493] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7d05e51ca25403e993415676dcaaf6e)
I20260620 01:50:18.276387 22693 raft_consensus.cc:3060] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:50:18.279699 22693 raft_consensus.cc:515] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:18.280706 22693 leader_election.cc:290] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [CANDIDATE]: Term 3 election: Requested vote from peers d7d05e51ca25403e993415676dcaaf6e (127.20.1.190:46541), 4f7675b441a14937960e9dbe0cae4ea8 (127.20.1.189:44845)
I20260620 01:50:18.281631 22040 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baaadc5c42084ad8b930708422b8fda6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4f7675b441a14937960e9dbe0cae4ea8"
I20260620 01:50:18.283064 22643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baaadc5c42084ad8b930708422b8fda6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7d05e51ca25403e993415676dcaaf6e"
I20260620 01:50:18.283418 22643 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:50:18.290897 22643 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baaadc5c42084ad8b930708422b8fda6 in term 3.
I20260620 01:50:18.291782 22100 leader_election.cc:304] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: baaadc5c42084ad8b930708422b8fda6, d7d05e51ca25403e993415676dcaaf6e; no voters: 4f7675b441a14937960e9dbe0cae4ea8
I20260620 01:50:18.292367 22693 raft_consensus.cc:2804] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 01:50:18.292929 22693 raft_consensus.cc:697] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [term 3 LEADER]: Becoming Leader. State: Replica: baaadc5c42084ad8b930708422b8fda6, State: Running, Role: LEADER
I20260620 01:50:18.293606 22693 consensus_queue.cc:237] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:18.298933 22695 sys_catalog.cc:455] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baaadc5c42084ad8b930708422b8fda6. Latest consensus state: current_term: 3 leader_uuid: "baaadc5c42084ad8b930708422b8fda6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:18.299522 22695 sys_catalog.cc:458] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:18.312952 22697 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:50:18.316921 22697 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:50:18.318686 22697 catalog_manager.cc:1294] Loaded cluster ID: e823db54e5164b59a86f95df26cb7cc5
I20260620 01:50:18.318975 22697 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:50:18.322383 22697 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:50:18.323911 22455 heartbeater.cc:344] Connected to a master server at 127.20.1.190:46541
I20260620 01:50:18.324221 22455 heartbeater.cc:499] Master 127.20.1.190:46541 was elected leader, sending a full tablet report...
I20260620 01:50:18.325248 22697 catalog_manager.cc:6080] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6: Loaded TSK: 0
I20260620 01:50:18.325817 22633 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8f13a66dd2e143e6b82c4a96325346c7" instance_seqno: 1781920216213231) as {username='slave'} at 127.20.1.130:39371; Asking this server to re-register.
I20260620 01:50:18.326977 22455 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:18.327245 22697 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:50:18.329115 22633 ts_manager.cc:194] Registered new tserver with Master: 8f13a66dd2e143e6b82c4a96325346c7 (127.20.1.130:36371)
W20260620 01:50:18.436620 22691 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:18.437083 22691 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:18.437214 22691 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:18.448312 22691 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:18.448562 22691 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:18.460434 22691 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:34385
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=43471
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:18.462504 22691 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:18.464805 22691 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:18.479770 22703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:18.480264 22706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:18.479784 22704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:18.481799 22691 server_base.cc:1061] running on GCE node
I20260620 01:50:18.482641 22691 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:18.482862 22691 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:18.482969 22691 hybrid_clock.cc:648] HybridClock initialized: now 1781920218482939 us; error 0 us; skew 500 ppm
I20260620 01:50:18.483395 22691 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:18.486155 22691 webserver.cc:533] Webserver started at http://127.20.1.129:43471/ using document root <none> and password file <none>
I20260620 01:50:18.487197 22691 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:18.487367 22691 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:18.496201 22691 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 01:50:18.501483 22711 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:18.504024 22691 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:50:18.504272 22691 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "b377b8e640814c82a541b0847e881666"
format_stamp: "Formatted at 2026-06-20 01:50:06 on dist-test-slave-wg14"
I20260620 01:50:18.505280 22691 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:18.541925 22691 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:18.543116 22691 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:18.543538 22691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:18.545100 22691 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:18.547418 22691 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:18.547544 22691 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:18.547683 22691 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:18.547736 22691 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:18.591044 22691 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:34385
I20260620 01:50:18.591071 22823 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:34385 every 8 connection(s)
I20260620 01:50:18.594180 22691 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:18.604568 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 22691
I20260620 01:50:18.605049 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 22691
I20260620 01:50:18.650252 22826 heartbeater.cc:344] Connected to a master server at 127.20.1.189:44845
I20260620 01:50:18.651015 22826 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:18.650338 22824 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44117
I20260620 01:50:18.651393 22824 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:18.652891 22824 heartbeater.cc:507] Master 127.20.1.188:44117 requested a full tablet report, sending...
I20260620 01:50:18.653046 22826 heartbeater.cc:507] Master 127.20.1.189:44845 requested a full tablet report, sending...
I20260620 01:50:18.653685 22819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:18.655921 22030 ts_manager.cc:194] Re-registered known tserver with Master: b377b8e640814c82a541b0847e881666 (127.20.1.129:34385)
I20260620 01:50:18.655921 22114 ts_manager.cc:194] Re-registered known tserver with Master: b377b8e640814c82a541b0847e881666 (127.20.1.129:34385)
I20260620 01:50:18.659538 22114 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:38125
I20260620 01:50:18.666977 22836 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7d05e51ca25403e993415676dcaaf6e)
I20260620 01:50:18.667412 22836 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } }
I20260620 01:50:18.670318 22836 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7d05e51ca25403e993415676dcaaf6e (127.20.1.190:46541), baaadc5c42084ad8b930708422b8fda6 (127.20.1.188:44117)
I20260620 01:50:18.678893 22643 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [term 3 FOLLOWER]: Refusing update from remote peer baaadc5c42084ad8b930708422b8fda6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:50:18.680492 22695 consensus_queue.cc:1048] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:50:18.682008 22124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7675b441a14937960e9dbe0cae4ea8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "baaadc5c42084ad8b930708422b8fda6" is_pre_election: true
I20260620 01:50:18.683066 22643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7675b441a14937960e9dbe0cae4ea8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7d05e51ca25403e993415676dcaaf6e" is_pre_election: true
I20260620 01:50:18.683897 22016 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f7675b441a14937960e9dbe0cae4ea8; no voters: baaadc5c42084ad8b930708422b8fda6, d7d05e51ca25403e993415676dcaaf6e
I20260620 01:50:18.684381 22836 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 01:50:18.688954 22839 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader baaadc5c42084ad8b930708422b8fda6. Latest consensus state: current_term: 3 leader_uuid: "baaadc5c42084ad8b930708422b8fda6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:18.689345 22839 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:18.691875 22693 sys_catalog.cc:455] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "baaadc5c42084ad8b930708422b8fda6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:18.692356 22693 sys_catalog.cc:458] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:18.693734 22040 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:50:18.698856 22040 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [term 3 FOLLOWER]: Refusing update from remote peer baaadc5c42084ad8b930708422b8fda6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:50:18.700073 22693 consensus_queue.cc:1048] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:50:18.707098 22839 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "baaadc5c42084ad8b930708422b8fda6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:18.707420 22839 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d05e51ca25403e993415676dcaaf6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:18.708410 22846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "baaadc5c42084ad8b930708422b8fda6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:18.708791 22846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:18.709105 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baaadc5c42084ad8b930708422b8fda6. Latest consensus state: current_term: 3 leader_uuid: "baaadc5c42084ad8b930708422b8fda6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:18.709368 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7675b441a14937960e9dbe0cae4ea8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:18.771826 22825 heartbeater.cc:344] Connected to a master server at 127.20.1.190:46541
I20260620 01:50:18.772104 22825 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:18.772833 22825 heartbeater.cc:507] Master 127.20.1.190:46541 requested a full tablet report, sending...
I20260620 01:50:18.773474 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 22691
I20260620 01:50:18.790969 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:34385
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--webserver_port=43471
--webserver_interface=127.20.1.129
--time_source=system_unsync
--fault_before_start=1.0 with env {}
I20260620 01:50:19.237417 22695 sys_catalog.cc:455] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "baaadc5c42084ad8b930708422b8fda6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d05e51ca25403e993415676dcaaf6e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46541 } } peers { permanent_uuid: "4f7675b441a14937960e9dbe0cae4ea8" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44845 } } peers { permanent_uuid: "baaadc5c42084ad8b930708422b8fda6" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44117 } } }
I20260620 01:50:19.237739 22695 sys_catalog.cc:458] T 00000000000000000000000000000000 P baaadc5c42084ad8b930708422b8fda6 [sys.catalog]: This master's current role is: LEADER
W20260620 01:50:19.277065 22847 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:19.277817 22847 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:19.277933 22847 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:50:19.277997 22847 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:19.289404 22847 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:19.289862 22847 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:19.303428 22847 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:34385
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=43471
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:19.305807 22847 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:19.308341 22847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:19.323498 22855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:19.323660 22856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:19.326927 22858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:19.330129 22847 server_base.cc:1061] running on GCE node
I20260620 01:50:19.331110 22847 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:19.331336 22847 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:19.331437 22847 hybrid_clock.cc:648] HybridClock initialized: now 1781920219331394 us; error 0 us; skew 500 ppm
I20260620 01:50:19.331833 22847 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:19.336992 22847 webserver.cc:533] Webserver started at http://127.20.1.129:43471/ using document root <none> and password file <none>
I20260620 01:50:19.338064 22847 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:19.338214 22847 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:19.347806 22847 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 01:50:19.352560 22863 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:19.355051 22847 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:50:19.355358 22847 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "b377b8e640814c82a541b0847e881666"
format_stamp: "Formatted at 2026-06-20 01:50:06 on dist-test-slave-wg14"
I20260620 01:50:19.356737 22847 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:19.388711 22847 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:19.389832 22847 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:19.390345 22847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:19.392066 22847 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:19.394950 22847 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:19.395176 22847 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:19.395318 22847 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:19.395552 22847 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:50:19.396307 22847 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:50:19.396427 22847 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:50:19.396489 22847 fault_injection.cc:59] injecting fault for kudu (pid 22847): FLAGS_fault_before_start (process will exit)
I20260620 01:50:19.413772 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:34385
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--webserver_port=43471
--webserver_interface=127.20.1.129
--time_source=system_unsync with env {}
W20260620 01:50:19.868276 22871 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:19.868687 22871 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:19.868821 22871 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:19.880849 22871 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:19.881148 22871 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:19.894598 22871 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:34385
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=43471
--tserver_master_addrs=127.20.1.190:46541,127.20.1.189:44845,127.20.1.188:44117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:19.897010 22871 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:19.899554 22871 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:19.914625 22877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:19.914649 22876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:19.914625 22879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:19.916226 22871 server_base.cc:1061] running on GCE node
I20260620 01:50:19.917074 22871 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:50:19.917277 22871 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:50:19.917368 22871 hybrid_clock.cc:648] HybridClock initialized: now 1781920219917344 us; error 0 us; skew 500 ppm
I20260620 01:50:19.917690 22871 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:19.921627 22871 webserver.cc:533] Webserver started at http://127.20.1.129:43471/ using document root <none> and password file <none>
I20260620 01:50:19.922955 22871 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:19.923116 22871 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:19.931330 22871 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.002s
I20260620 01:50:19.936232 22884 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:19.940130 22871 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.004s
I20260620 01:50:19.940474 22871 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "b377b8e640814c82a541b0847e881666"
format_stamp: "Formatted at 2026-06-20 01:50:06 on dist-test-slave-wg14"
I20260620 01:50:19.941587 22871 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:19.965508 22871 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:19.966737 22871 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:19.967142 22871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:19.968525 22871 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:19.970819 22871 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:19.970952 22871 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:19.971196 22871 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:19.971328 22871 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 01:50:20.008033 22590 heartbeater.cc:499] Master 127.20.1.188:44117 was elected leader, sending a full tablet report...
I20260620 01:50:20.045529 22871 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:34385
I20260620 01:50:20.045578 22996 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:34385 every 8 connection(s)
I20260620 01:50:20.047608 22871 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:20.057639 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 22871
I20260620 01:50:20.058339 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 22871
I20260620 01:50:20.099169 22998 heartbeater.cc:344] Connected to a master server at 127.20.1.190:46541
I20260620 01:50:20.099211 22997 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44117
I20260620 01:50:20.101502 22998 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:20.101502 22997 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:20.101956 22992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:20.101492 22999 heartbeater.cc:344] Connected to a master server at 127.20.1.189:44845
I20260620 01:50:20.103524 22999 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:20.222010 22997 heartbeater.cc:507] Master 127.20.1.188:44117 requested a full tablet report, sending...
I20260620 01:50:20.222232 22999 heartbeater.cc:507] Master 127.20.1.189:44845 requested a full tablet report, sending...
I20260620 01:50:20.222278 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 22871
I20260620 01:50:20.241202 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 22324
I20260620 01:50:20.254477 22449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:20.360500 22454 heartbeater.cc:499] Master 127.20.1.188:44117 was elected leader, sending a full tablet report...
I20260620 01:50:20.363174 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 22324
W20260620 01:50:20.379783 22016 connection.cc:573] server connection from 127.20.1.130:40655 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 01:50:20.379781 22101 connection.cc:573] server connection from 127.20.1.130:41783 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 01:50:20.379781 22617 connection.cc:573] server connection from 127.20.1.130:39371 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:50:20.380776 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 22464
I20260620 01:50:20.393343 22585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:20.507519 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 22464
I20260620 01:50:20.523089 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 22601
I20260620 01:50:20.534372 22663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:20.671286 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 22601
I20260620 01:50:20.692063 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 21984
I20260620 01:50:20.694208 22060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:20.828279 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 21984
I20260620 01:50:20.852449 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 22067
I20260620 01:50:20.854071 22144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:20.980304 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 22067
[ OK ] ExternalMiniClusterTest.TestBasicOperation/0 (18044 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/1
2026-06-20T01:50:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:50:21Z Disabled control of system clock
I20260620 01:50:21.059433 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44517
--webserver_interface=127.20.1.190
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077 with env {}
W20260620 01:50:21.530593 23020 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:21.531055 23020 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:21.531180 23020 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:21.542200 23020 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:21.542348 23020 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:21.542410 23020 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:21.542452 23020 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:21.553963 23020 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:21.555975 23020 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:21.558336 23020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:21.571368 23026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:21.571516 23025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:21.573510 23028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:21.573992 23020 server_base.cc:1061] running on GCE node
I20260620 01:50:21.575109 23020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:21.577183 23020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:21.578631 23020 hybrid_clock.cc:648] HybridClock initialized: now 1781920221578511 us; error 68 us; skew 500 ppm
I20260620 01:50:21.579480 23020 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:21.583844 23020 webserver.cc:533] Webserver started at http://127.20.1.190:46205/ using document root <none> and password file <none>
I20260620 01:50:21.584887 23020 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:21.585026 23020 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:21.585793 23020 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:21.589488 23020 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "94716efbcd394c87b6b422791fce713d"
format_stamp: "Formatted at 2026-06-20 01:50:21 on dist-test-slave-wg14"
I20260620 01:50:21.590799 23020 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "94716efbcd394c87b6b422791fce713d"
format_stamp: "Formatted at 2026-06-20 01:50:21 on dist-test-slave-wg14"
I20260620 01:50:21.598948 23020 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260620 01:50:21.609247 23034 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:21.612386 23020 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260620 01:50:21.612695 23020 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "94716efbcd394c87b6b422791fce713d"
format_stamp: "Formatted at 2026-06-20 01:50:21 on dist-test-slave-wg14"
I20260620 01:50:21.612989 23020 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:21.638679 23020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:21.639858 23020 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:21.640234 23020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:21.669631 23020 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44517
I20260620 01:50:21.669706 23085 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44517 every 8 connection(s)
I20260620 01:50:21.672152 23020 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:50:21.679711 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 23020
I20260620 01:50:21.679999 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:50:21.680539 23086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:50:21.681668 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:45519
--webserver_interface=127.20.1.189
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077 with env {}
I20260620 01:50:21.691679 23086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:21.717015 23086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:50:21.719193 23036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45519: connect: Connection refused (error 111)
W20260620 01:50:21.721434 23086 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:45519: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45519: connect: Connection refused (error 111)
I20260620 01:50:21.770921 23086 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } attempt: 1
W20260620 01:50:21.773247 23086 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:45519: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45519: connect: Connection refused (error 111)
I20260620 01:50:21.841593 23086 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } attempt: 2
W20260620 01:50:21.844497 23086 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:45519: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45519: connect: Connection refused (error 111)
I20260620 01:50:21.935858 23086 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } attempt: 3
W20260620 01:50:21.938774 23086 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:45519: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45519: connect: Connection refused (error 111)
I20260620 01:50:22.082194 23086 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } attempt: 4
W20260620 01:50:22.084832 23086 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:45519: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45519: connect: Connection refused (error 111)
W20260620 01:50:22.149952 23088 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:22.150429 23088 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:22.150540 23088 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:22.161653 23088 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:22.161782 23088 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:22.161870 23088 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:22.161933 23088 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:22.174376 23088 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:45519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:22.176627 23088 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:22.179487 23088 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:22.192414 23097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:22.193554 23099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:22.193485 23096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:22.194662 23088 server_base.cc:1061] running on GCE node
I20260620 01:50:22.195730 23088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:22.197847 23088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:22.199447 23088 hybrid_clock.cc:648] HybridClock initialized: now 1781920222199406 us; error 63 us; skew 500 ppm
I20260620 01:50:22.199967 23088 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:22.205577 23088 webserver.cc:533] Webserver started at http://127.20.1.189:36055/ using document root <none> and password file <none>
I20260620 01:50:22.206561 23088 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:22.206709 23088 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:22.207110 23088 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:22.210026 23088 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "2d65d5070b2347bf8f6f12e6b1865c9a"
format_stamp: "Formatted at 2026-06-20 01:50:22 on dist-test-slave-wg14"
I20260620 01:50:22.211113 23088 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "2d65d5070b2347bf8f6f12e6b1865c9a"
format_stamp: "Formatted at 2026-06-20 01:50:22 on dist-test-slave-wg14"
I20260620 01:50:22.221576 23088 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.008s sys 0.000s
I20260620 01:50:22.227579 23105 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:22.230129 23088 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.002s
I20260620 01:50:22.230427 23088 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "2d65d5070b2347bf8f6f12e6b1865c9a"
format_stamp: "Formatted at 2026-06-20 01:50:22 on dist-test-slave-wg14"
I20260620 01:50:22.230706 23088 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:22.269659 23088 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:22.270684 23088 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:22.271211 23088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:22.302323 23088 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:45519
I20260620 01:50:22.302350 23156 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:45519 every 8 connection(s)
I20260620 01:50:22.304435 23088 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:50:22.310202 23157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:50:22.314534 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 23088
I20260620 01:50:22.314815 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:50:22.316742 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:34077
--webserver_interface=127.20.1.188
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077 with env {}
I20260620 01:50:22.318027 23157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:22.340206 23157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:22.353786 23157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:50:22.355473 23107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34077: connect: Connection refused (error 111)
W20260620 01:50:22.357990 23157 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34077: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34077: connect: Connection refused (error 111)
I20260620 01:50:22.384157 23086 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } attempt: 5
I20260620 01:50:22.394522 23086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:50:22.397841 23086 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34077: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34077: connect: Connection refused (error 111)
I20260620 01:50:22.407486 23157 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } attempt: 1
W20260620 01:50:22.412712 23157 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34077: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34077: connect: Connection refused (error 111)
I20260620 01:50:22.449198 23086 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } attempt: 1
W20260620 01:50:22.452085 23086 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34077: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34077: connect: Connection refused (error 111)
I20260620 01:50:22.481037 23157 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } attempt: 2
W20260620 01:50:22.485880 23157 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34077: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34077: connect: Connection refused (error 111)
I20260620 01:50:22.520535 23086 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } attempt: 2
W20260620 01:50:22.523526 23086 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34077: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34077: connect: Connection refused (error 111)
I20260620 01:50:22.577227 23157 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } attempt: 3
W20260620 01:50:22.580971 23157 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34077: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34077: connect: Connection refused (error 111)
I20260620 01:50:22.629859 23086 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } attempt: 3
W20260620 01:50:22.632718 23086 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34077: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34077: connect: Connection refused (error 111)
I20260620 01:50:22.724315 23157 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } attempt: 4
W20260620 01:50:22.727615 23157 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34077: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34077: connect: Connection refused (error 111)
I20260620 01:50:22.810047 23086 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } attempt: 4
W20260620 01:50:22.812463 23086 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34077: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34077: connect: Connection refused (error 111)
W20260620 01:50:22.842571 23159 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:22.843051 23159 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:22.843226 23159 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:22.853179 23159 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:22.853348 23159 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:22.853452 23159 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:22.853525 23159 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:22.865685 23159 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:34077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:22.868079 23159 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:22.870986 23159 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:22.883828 23169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:22.883749 23168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:22.885586 23171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:22.885859 23159 server_base.cc:1061] running on GCE node
I20260620 01:50:22.886889 23159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:22.888746 23159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:22.890347 23159 hybrid_clock.cc:648] HybridClock initialized: now 1781920222890173 us; error 161 us; skew 500 ppm
I20260620 01:50:22.891100 23159 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:22.895025 23159 webserver.cc:533] Webserver started at http://127.20.1.188:34319/ using document root <none> and password file <none>
I20260620 01:50:22.896203 23159 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:22.896353 23159 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:22.896801 23159 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:22.900782 23159 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "5043f03c94574e408482f1e346500ae2"
format_stamp: "Formatted at 2026-06-20 01:50:22 on dist-test-slave-wg14"
I20260620 01:50:22.901875 23159 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "5043f03c94574e408482f1e346500ae2"
format_stamp: "Formatted at 2026-06-20 01:50:22 on dist-test-slave-wg14"
I20260620 01:50:22.909211 23159 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260620 01:50:22.914445 23177 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:22.916911 23159 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260620 01:50:22.917222 23159 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "5043f03c94574e408482f1e346500ae2"
format_stamp: "Formatted at 2026-06-20 01:50:22 on dist-test-slave-wg14"
I20260620 01:50:22.917526 23159 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:22.947901 23159 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:22.949092 23159 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:22.949537 23159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:22.980566 23228 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:34077 every 8 connection(s)
I20260620 01:50:22.980558 23159 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:34077
I20260620 01:50:22.982774 23159 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:50:22.984019 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 23159
I20260620 01:50:22.984673 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:50:22.989943 23229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:50:23.000849 23229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:23.020183 23229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:23.026889 23157 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } attempt: 5
I20260620 01:50:23.028585 23229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:23.042563 23157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a: Bootstrap starting.
I20260620 01:50:23.043797 23229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2: Bootstrap starting.
I20260620 01:50:23.047353 23157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a: Neither blocks nor log segments found. Creating new log.
I20260620 01:50:23.047569 23229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2: Neither blocks nor log segments found. Creating new log.
I20260620 01:50:23.048874 23157 log.cc:826] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:23.048877 23229 log.cc:826] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:23.053908 23229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2: No bootstrap required, opened a new log
I20260620 01:50:23.053916 23157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a: No bootstrap required, opened a new log
I20260620 01:50:23.063755 23157 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:23.063755 23229 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:23.064215 23157 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:50:23.064215 23229 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:50:23.064373 23229 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5043f03c94574e408482f1e346500ae2, State: Initialized, Role: FOLLOWER
I20260620 01:50:23.064394 23157 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d65d5070b2347bf8f6f12e6b1865c9a, State: Initialized, Role: FOLLOWER
I20260620 01:50:23.065457 23157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:23.065423 23229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:23.068181 23239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:23.068361 23240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:23.068675 23240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:23.068810 23239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:23.069738 23229 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:23.069738 23157 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:23.082031 23159 master_runner.cc:187] Error getting master registration for 127.20.1.190:44517: OK, instance_id { permanent_uuid: "94716efbcd394c87b6b422791fce713d" instance_seqno: 1781920221660087 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:50:23.089183 23159 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:50:23.081058 23088 master_runner.cc:187] Error getting master registration for 127.20.1.190:44517: OK, instance_id { permanent_uuid: "94716efbcd394c87b6b422791fce713d" instance_seqno: 1781920221660087 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:50:23.089851 23088 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:50:23.089843 23086 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } attempt: 5
W20260620 01:50:23.094877 23262 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:50:23.097635 23262 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 01:50:23.099895 23261 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:50:23.100150 23261 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:50:23.107941 23086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: Bootstrap starting.
I20260620 01:50:23.112790 23086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: Neither blocks nor log segments found. Creating new log.
I20260620 01:50:23.114187 23086 log.cc:826] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:23.117604 23086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: No bootstrap required, opened a new log
I20260620 01:50:23.126072 23086 raft_consensus.cc:359] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:23.126580 23086 raft_consensus.cc:385] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:50:23.126736 23086 raft_consensus.cc:740] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94716efbcd394c87b6b422791fce713d, State: Initialized, Role: FOLLOWER
I20260620 01:50:23.128052 23086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:23.131031 23239 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:50:23.131445 23239 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:23.131402 23266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:23.131842 23266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:23.133317 23086 sys_catalog.cc:565] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:23.133872 23239 leader_election.cc:290] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94716efbcd394c87b6b422791fce713d (127.20.1.190:44517), 2d65d5070b2347bf8f6f12e6b1865c9a (127.20.1.189:45519)
I20260620 01:50:23.135082 23061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5043f03c94574e408482f1e346500ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94716efbcd394c87b6b422791fce713d" is_pre_election: true
I20260620 01:50:23.135898 23132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5043f03c94574e408482f1e346500ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" is_pre_election: true
I20260620 01:50:23.136564 23061 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5043f03c94574e408482f1e346500ae2 in term 0.
I20260620 01:50:23.136605 23132 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5043f03c94574e408482f1e346500ae2 in term 0.
I20260620 01:50:23.137832 23180 leader_election.cc:304] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5043f03c94574e408482f1e346500ae2, 94716efbcd394c87b6b422791fce713d; no voters:
I20260620 01:50:23.138618 23239 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:50:23.138921 23239 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:50:23.139050 23239 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:50:23.143435 23239 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
W20260620 01:50:23.144644 23040 tablet.cc:2406] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:50:23.145151 23061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5043f03c94574e408482f1e346500ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94716efbcd394c87b6b422791fce713d"
I20260620 01:50:23.145480 23061 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:50:23.149482 23061 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5043f03c94574e408482f1e346500ae2 in term 1.
I20260620 01:50:23.150429 23180 leader_election.cc:304] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5043f03c94574e408482f1e346500ae2, 94716efbcd394c87b6b422791fce713d; no voters:
I20260620 01:50:23.155377 23132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5043f03c94574e408482f1e346500ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a"
I20260620 01:50:23.155668 23132 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:50:23.161064 23132 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5043f03c94574e408482f1e346500ae2 in term 1.
I20260620 01:50:23.162846 23239 leader_election.cc:290] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [CANDIDATE]: Term 1 election: Requested vote from peers 94716efbcd394c87b6b422791fce713d (127.20.1.190:44517), 2d65d5070b2347bf8f6f12e6b1865c9a (127.20.1.189:45519)
I20260620 01:50:23.163311 23239 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:50:23.163513 23239 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 1 LEADER]: Becoming Leader. State: Replica: 5043f03c94574e408482f1e346500ae2, State: Running, Role: LEADER
I20260620 01:50:23.164186 23239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:23.183260 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5043f03c94574e408482f1e346500ae2. Latest consensus state: current_term: 1 leader_uuid: "5043f03c94574e408482f1e346500ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:23.183789 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:23.195333 23272 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:50:23.198979 23272 catalog_manager.cc:1519] Initializing Kudu cluster ID...
W20260620 01:50:23.206772 23183 tablet.cc:2406] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:50:23.228680 23132 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 1 FOLLOWER]: Refusing update from remote peer 5043f03c94574e408482f1e346500ae2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:50:23.230397 23268 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:50:23.233318 23061 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 1 FOLLOWER]: Refusing update from remote peer 5043f03c94574e408482f1e346500ae2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:50:23.237257 23268 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 01:50:23.275106 23111 tablet.cc:2406] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:50:23.275838 23266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5043f03c94574e408482f1e346500ae2. Latest consensus state: current_term: 1 leader_uuid: "5043f03c94574e408482f1e346500ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:23.276191 23266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:23.281369 23284 mvcc.cc:204] Tried to move back new op lower bound from 7298745234320171008 to 7298745234100080640. Current Snapshot: MvccSnapshot[applied={T|T < 7298745234320171008}]
W20260620 01:50:23.291885 23281 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:50:23.292112 23281 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:50:23.295898 23266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5043f03c94574e408482f1e346500ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:23.296195 23266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:23.301985 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5043f03c94574e408482f1e346500ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:23.302330 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:23.303870 23240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5043f03c94574e408482f1e346500ae2. Latest consensus state: current_term: 1 leader_uuid: "5043f03c94574e408482f1e346500ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:23.304212 23240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:23.316395 23272 catalog_manager.cc:1382] Generated new cluster ID: 588c15584dca4915a1945314a0077392
I20260620 01:50:23.316644 23272 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:50:23.320415 23240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5043f03c94574e408482f1e346500ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:23.321007 23240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:23.322995 23283 mvcc.cc:204] Tried to move back new op lower bound from 7298745234320171008 to 7298745234100080640. Current Snapshot: MvccSnapshot[applied={T|T < 7298745234320171008 or (T in {7298745234320171008})}]
I20260620 01:50:23.331866 23239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5043f03c94574e408482f1e346500ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:23.332887 23239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:23.395157 23272 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:50:23.397145 23272 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:50:23.415866 23272 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2: Generated new TSK 0
I20260620 01:50:23.417382 23272 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:50:23.425350 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--builtin_ntp_servers=127.20.1.148:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:50:24.101493 23262 catalog_manager.cc:1294] Loaded cluster ID: 588c15584dca4915a1945314a0077392
I20260620 01:50:24.101625 23262 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a: loading cluster ID for follower catalog manager: success
I20260620 01:50:24.105029 23262 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a: acquiring CA information for follower catalog manager: success
I20260620 01:50:24.107522 23262 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 01:50:24.164359 23292 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:24.165045 23292 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:24.165200 23292 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:24.178673 23292 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:24.178879 23292 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:24.194700 23292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:24.197624 23292 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:24.201937 23292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:24.215602 23298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:24.216225 23300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:24.215637 23297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:24.218319 23292 server_base.cc:1061] running on GCE node
I20260620 01:50:24.219254 23292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:24.220784 23292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:24.222314 23292 hybrid_clock.cc:648] HybridClock initialized: now 1781920224222232 us; error 84 us; skew 500 ppm
I20260620 01:50:24.222810 23292 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:24.226115 23292 webserver.cc:533] Webserver started at http://127.20.1.129:40657/ using document root <none> and password file <none>
I20260620 01:50:24.227837 23292 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:24.228061 23292 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:24.228830 23292 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:24.233042 23292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "54bb296f47a247b5b75d37f7f4c5117d"
format_stamp: "Formatted at 2026-06-20 01:50:24 on dist-test-slave-wg14"
I20260620 01:50:24.235265 23292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "54bb296f47a247b5b75d37f7f4c5117d"
format_stamp: "Formatted at 2026-06-20 01:50:24 on dist-test-slave-wg14"
I20260620 01:50:24.244884 23292 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260620 01:50:24.249846 23306 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:24.253741 23292 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260620 01:50:24.254074 23292 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "54bb296f47a247b5b75d37f7f4c5117d"
format_stamp: "Formatted at 2026-06-20 01:50:24 on dist-test-slave-wg14"
I20260620 01:50:24.254412 23292 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:24.281446 23292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:24.282636 23292 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:24.283017 23292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:24.285205 23292 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:24.287417 23292 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:24.287547 23292 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:24.287649 23292 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:24.287734 23292 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:24.296798 23281 catalog_manager.cc:1294] Loaded cluster ID: 588c15584dca4915a1945314a0077392
I20260620 01:50:24.297034 23281 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: loading cluster ID for follower catalog manager: success
I20260620 01:50:24.304800 23281 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: acquiring CA information for follower catalog manager: success
I20260620 01:50:24.307482 23281 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:50:24.358173 23292 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:45333
I20260620 01:50:24.358218 23418 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:45333 every 8 connection(s)
I20260620 01:50:24.361129 23292 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:24.365548 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 23292
I20260620 01:50:24.365770 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:50:24.373714 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--builtin_ntp_servers=127.20.1.148:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:50:24.395607 23421 heartbeater.cc:344] Connected to a master server at 127.20.1.189:45519
I20260620 01:50:24.395620 23420 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44517
I20260620 01:50:24.396416 23420 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:24.397359 23419 heartbeater.cc:344] Connected to a master server at 127.20.1.188:34077
I20260620 01:50:24.398561 23419 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:24.396420 23421 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:24.399688 23420 heartbeater.cc:507] Master 127.20.1.190:44517 requested a full tablet report, sending...
I20260620 01:50:24.399772 23419 heartbeater.cc:507] Master 127.20.1.188:34077 requested a full tablet report, sending...
I20260620 01:50:24.400612 23421 heartbeater.cc:507] Master 127.20.1.189:45519 requested a full tablet report, sending...
I20260620 01:50:24.403640 23051 ts_manager.cc:194] Registered new tserver with Master: 54bb296f47a247b5b75d37f7f4c5117d (127.20.1.129:45333)
I20260620 01:50:24.403640 23122 ts_manager.cc:194] Registered new tserver with Master: 54bb296f47a247b5b75d37f7f4c5117d (127.20.1.129:45333)
I20260620 01:50:24.403641 23194 ts_manager.cc:194] Registered new tserver with Master: 54bb296f47a247b5b75d37f7f4c5117d (127.20.1.129:45333)
I20260620 01:50:24.406205 23194 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:38295
W20260620 01:50:24.843375 23429 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:24.843762 23429 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:24.843851 23429 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:24.853468 23429 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:24.853749 23429 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:50:24.867909 23429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:24.870563 23429 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:24.873212 23429 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:24.887058 23435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:24.887171 23434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:24.887840 23437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:24.888329 23429 server_base.cc:1061] running on GCE node
I20260620 01:50:24.889104 23429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:24.890669 23429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:24.892184 23429 hybrid_clock.cc:648] HybridClock initialized: now 1781920224892107 us; error 58 us; skew 500 ppm
I20260620 01:50:24.892699 23429 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:24.896106 23429 webserver.cc:533] Webserver started at http://127.20.1.130:38677/ using document root <none> and password file <none>
I20260620 01:50:24.897294 23429 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:24.897572 23429 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:24.898155 23429 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:24.901938 23429 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/data/instance:
uuid: "e52eb4213cfc4dc6aaa391a9ec00de90"
format_stamp: "Formatted at 2026-06-20 01:50:24 on dist-test-slave-wg14"
I20260620 01:50:24.902998 23429 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance:
uuid: "e52eb4213cfc4dc6aaa391a9ec00de90"
format_stamp: "Formatted at 2026-06-20 01:50:24 on dist-test-slave-wg14"
I20260620 01:50:24.909948 23429 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260620 01:50:24.914951 23443 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:24.917714 23429 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:50:24.918025 23429 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "e52eb4213cfc4dc6aaa391a9ec00de90"
format_stamp: "Formatted at 2026-06-20 01:50:24 on dist-test-slave-wg14"
I20260620 01:50:24.918699 23429 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:24.938405 23429 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:24.939416 23429 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:24.939673 23429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:24.940897 23429 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:24.943221 23429 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:24.943343 23429 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:24.943437 23429 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:24.943499 23429 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:24.986768 23429 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:42321
I20260620 01:50:24.986799 23555 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:42321 every 8 connection(s)
I20260620 01:50:24.988680 23429 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:50:24.993662 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 23429
I20260620 01:50:24.994002 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
I20260620 01:50:25.012681 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--builtin_ntp_servers=127.20.1.148:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:50:25.022918 23558 heartbeater.cc:344] Connected to a master server at 127.20.1.189:45519
I20260620 01:50:25.022965 23557 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44517
I20260620 01:50:25.023000 23556 heartbeater.cc:344] Connected to a master server at 127.20.1.188:34077
I20260620 01:50:25.023653 23557 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:25.023656 23558 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:25.023655 23556 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:25.025405 23557 heartbeater.cc:507] Master 127.20.1.190:44517 requested a full tablet report, sending...
I20260620 01:50:25.025609 23556 heartbeater.cc:507] Master 127.20.1.188:34077 requested a full tablet report, sending...
I20260620 01:50:25.025460 23558 heartbeater.cc:507] Master 127.20.1.189:45519 requested a full tablet report, sending...
I20260620 01:50:25.027864 23194 ts_manager.cc:194] Registered new tserver with Master: e52eb4213cfc4dc6aaa391a9ec00de90 (127.20.1.130:42321)
I20260620 01:50:25.027925 23051 ts_manager.cc:194] Registered new tserver with Master: e52eb4213cfc4dc6aaa391a9ec00de90 (127.20.1.130:42321)
I20260620 01:50:25.028100 23122 ts_manager.cc:194] Registered new tserver with Master: e52eb4213cfc4dc6aaa391a9ec00de90 (127.20.1.130:42321)
I20260620 01:50:25.029055 23194 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.130:58731
I20260620 01:50:25.411799 23419 heartbeater.cc:499] Master 127.20.1.188:34077 was elected leader, sending a full tablet report...
W20260620 01:50:25.444099 23566 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:25.444610 23566 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:25.444751 23566 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:25.454556 23566 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:25.454748 23566 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:50:25.466172 23566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:25.468297 23566 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:25.471212 23566 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:25.487069 23574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:25.487071 23572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:25.487030 23571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:25.488202 23566 server_base.cc:1061] running on GCE node
I20260620 01:50:25.489019 23566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:25.490291 23566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:25.493775 23566 hybrid_clock.cc:648] HybridClock initialized: now 1781920225493659 us; error 77 us; skew 500 ppm
I20260620 01:50:25.494284 23566 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:25.498133 23566 webserver.cc:533] Webserver started at http://127.20.1.131:34463/ using document root <none> and password file <none>
I20260620 01:50:25.499238 23566 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:25.499399 23566 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:25.499833 23566 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:25.503005 23566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/data/instance:
uuid: "78bdd4be122c41ad9cd7f732127cb11d"
format_stamp: "Formatted at 2026-06-20 01:50:25 on dist-test-slave-wg14"
I20260620 01:50:25.504357 23566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance:
uuid: "78bdd4be122c41ad9cd7f732127cb11d"
format_stamp: "Formatted at 2026-06-20 01:50:25 on dist-test-slave-wg14"
I20260620 01:50:25.511479 23566 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 01:50:25.516507 23580 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:25.518884 23566 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:50:25.519250 23566 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "78bdd4be122c41ad9cd7f732127cb11d"
format_stamp: "Formatted at 2026-06-20 01:50:25 on dist-test-slave-wg14"
I20260620 01:50:25.519564 23566 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:25.548982 23566 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:25.550073 23566 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:25.550678 23566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:25.552368 23566 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:25.555696 23566 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:25.555883 23566 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:25.556022 23566 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:25.556100 23566 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:25.602105 23566 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:44213
I20260620 01:50:25.602157 23692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:44213 every 8 connection(s)
I20260620 01:50:25.604753 23566 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:50:25.606383 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 23566
I20260620 01:50:25.606635 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
I20260620 01:50:25.634907 23694 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44517
I20260620 01:50:25.635005 23693 heartbeater.cc:344] Connected to a master server at 127.20.1.188:34077
I20260620 01:50:25.635890 23693 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:25.636297 23695 heartbeater.cc:344] Connected to a master server at 127.20.1.189:45519
I20260620 01:50:25.636504 23695 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:25.635890 23694 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:25.637499 23693 heartbeater.cc:507] Master 127.20.1.188:34077 requested a full tablet report, sending...
I20260620 01:50:25.637566 23695 heartbeater.cc:507] Master 127.20.1.189:45519 requested a full tablet report, sending...
I20260620 01:50:25.638059 23694 heartbeater.cc:507] Master 127.20.1.190:44517 requested a full tablet report, sending...
I20260620 01:50:25.639751 23122 ts_manager.cc:194] Registered new tserver with Master: 78bdd4be122c41ad9cd7f732127cb11d (127.20.1.131:44213)
I20260620 01:50:25.639772 23051 ts_manager.cc:194] Registered new tserver with Master: 78bdd4be122c41ad9cd7f732127cb11d (127.20.1.131:44213)
I20260620 01:50:25.641476 23194 ts_manager.cc:194] Registered new tserver with Master: 78bdd4be122c41ad9cd7f732127cb11d (127.20.1.131:44213)
I20260620 01:50:25.642541 23194 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.131:55419
I20260620 01:50:25.659222 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:50:25.673590 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:50:25.674024 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 23020
I20260620 01:50:25.675582 23081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:25.793329 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 23020
W20260620 01:50:25.812865 23180 connection.cc:573] client connection to 127.20.1.190:44517 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 01:50:25.813230 23180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:50:25.814626 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44517
--webserver_interface=127.20.1.190
--webserver_port=46205
--builtin_ntp_servers=127.20.1.148:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077 with env {}
W20260620 01:50:25.816737 23180 consensus_peers.cc:597] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 -> Peer 94716efbcd394c87b6b422791fce713d (127.20.1.190:44517): Couldn't send request to peer 94716efbcd394c87b6b422791fce713d. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:44517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 01:50:26.032226 23556 heartbeater.cc:499] Master 127.20.1.188:34077 was elected leader, sending a full tablet report...
W20260620 01:50:26.034374 23557 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:44517 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:44517: connect: Connection refused (error 111)
W20260620 01:50:26.267915 23706 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:26.268337 23706 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:26.268458 23706 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:26.283037 23706 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:26.283242 23706 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:26.283332 23706 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:26.283406 23706 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:26.298101 23706 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=46205
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:26.300311 23706 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:26.303287 23706 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:26.316736 23712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:26.317142 23713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:26.318380 23715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:26.319835 23706 server_base.cc:1061] running on GCE node
I20260620 01:50:26.321862 23706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:26.324671 23706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:26.326300 23706 hybrid_clock.cc:648] HybridClock initialized: now 1781920226326131 us; error 149 us; skew 500 ppm
I20260620 01:50:26.326862 23706 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:26.330595 23706 webserver.cc:533] Webserver started at http://127.20.1.190:46205/ using document root <none> and password file <none>
I20260620 01:50:26.332039 23706 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:26.332252 23706 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:26.340147 23706 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 01:50:26.345837 23721 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:26.349622 23706 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.007s sys 0.000s
I20260620 01:50:26.349895 23706 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "94716efbcd394c87b6b422791fce713d"
format_stamp: "Formatted at 2026-06-20 01:50:21 on dist-test-slave-wg14"
I20260620 01:50:26.350885 23706 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:26.367993 23291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5043f03c94574e408482f1e346500ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:26.368386 23291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:26.382830 23706 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:26.385949 23706 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:26.386561 23706 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260620 01:50:26.410548 23420 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:44517 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:44517: connect: Connection refused (error 111)
I20260620 01:50:26.418390 23778 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44517 every 8 connection(s)
I20260620 01:50:26.418390 23706 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44517
I20260620 01:50:26.420828 23706 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:50:26.423879 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 23706
I20260620 01:50:26.424284 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 23292
I20260620 01:50:26.431425 23779 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:50:26.437567 23779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: Bootstrap starting.
I20260620 01:50:26.443109 23414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:26.464622 23779 log.cc:826] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:26.482384 23779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:50:26.483330 23779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: Bootstrap complete.
I20260620 01:50:26.495744 23779 raft_consensus.cc:359] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:26.497502 23779 raft_consensus.cc:740] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94716efbcd394c87b6b422791fce713d, State: Initialized, Role: FOLLOWER
I20260620 01:50:26.498936 23779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:26.500463 23785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:26.500921 23785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:26.502154 23779 sys_catalog.cc:565] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:26.519340 23796 catalog_manager.cc:1294] Loaded cluster ID: 588c15584dca4915a1945314a0077392
I20260620 01:50:26.519500 23796 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: loading cluster ID for follower catalog manager: success
I20260620 01:50:26.527748 23796 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: acquiring CA information for follower catalog manager: success
I20260620 01:50:26.535198 23796 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:50:26.584932 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 23292
I20260620 01:50:26.604867 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:45333
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--webserver_port=40657
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:50:26.645864 23693 heartbeater.cc:499] Master 127.20.1.188:34077 was elected leader, sending a full tablet report...
I20260620 01:50:26.658705 23743 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "78bdd4be122c41ad9cd7f732127cb11d" instance_seqno: 1781920225589761) as {username='slave'} at 127.20.1.131:46445; Asking this server to re-register.
I20260620 01:50:26.660017 23694 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:26.665979 23743 ts_manager.cc:194] Registered new tserver with Master: 78bdd4be122c41ad9cd7f732127cb11d (127.20.1.131:44213)
I20260620 01:50:26.790307 23785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5043f03c94574e408482f1e346500ae2. Latest consensus state: current_term: 1 leader_uuid: "5043f03c94574e408482f1e346500ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:26.790956 23785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:27.056391 23557 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44517
I20260620 01:50:27.058163 23743 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e52eb4213cfc4dc6aaa391a9ec00de90" instance_seqno: 1781920224976442) as {username='slave'} at 127.20.1.130:42331; Asking this server to re-register.
I20260620 01:50:27.058984 23557 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:27.060714 23743 ts_manager.cc:194] Registered new tserver with Master: e52eb4213cfc4dc6aaa391a9ec00de90 (127.20.1.130:42321)
W20260620 01:50:27.097118 23802 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:27.097553 23802 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:27.098155 23802 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:27.109422 23802 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:27.109705 23802 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:27.122711 23802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:45333
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=40657
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:27.125048 23802 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:27.127650 23802 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:27.142596 23810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:27.142607 23809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:27.143388 23812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:27.144505 23802 server_base.cc:1061] running on GCE node
I20260620 01:50:27.145277 23802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:27.147156 23802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:27.148712 23802 hybrid_clock.cc:648] HybridClock initialized: now 1781920227148595 us; error 111 us; skew 500 ppm
I20260620 01:50:27.149245 23802 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:27.155287 23802 webserver.cc:533] Webserver started at http://127.20.1.129:40657/ using document root <none> and password file <none>
I20260620 01:50:27.156579 23802 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:27.156773 23802 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:27.165622 23802 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 01:50:27.169947 23818 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:27.172786 23802 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:50:27.173022 23802 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "54bb296f47a247b5b75d37f7f4c5117d"
format_stamp: "Formatted at 2026-06-20 01:50:24 on dist-test-slave-wg14"
I20260620 01:50:27.173936 23802 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:27.193300 23802 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:27.194576 23802 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:27.195044 23802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:27.196825 23802 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:27.200299 23802 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:27.200484 23802 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:27.200629 23802 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:27.200690 23802 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:27.247920 23802 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:45333
I20260620 01:50:27.247960 23930 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:45333 every 8 connection(s)
I20260620 01:50:27.249794 23802 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:27.250306 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 23802
I20260620 01:50:27.251061 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 23802
I20260620 01:50:27.286865 23932 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44517
I20260620 01:50:27.286969 23933 heartbeater.cc:344] Connected to a master server at 127.20.1.189:45519
I20260620 01:50:27.287318 23931 heartbeater.cc:344] Connected to a master server at 127.20.1.188:34077
I20260620 01:50:27.287473 23933 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:27.287510 23931 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:27.287515 23932 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:27.289018 23932 heartbeater.cc:507] Master 127.20.1.190:44517 requested a full tablet report, sending...
I20260620 01:50:27.288975 23933 heartbeater.cc:507] Master 127.20.1.189:45519 requested a full tablet report, sending...
I20260620 01:50:27.289757 23931 heartbeater.cc:507] Master 127.20.1.188:34077 requested a full tablet report, sending...
I20260620 01:50:27.292409 23194 ts_manager.cc:194] Re-registered known tserver with Master: 54bb296f47a247b5b75d37f7f4c5117d (127.20.1.129:45333)
I20260620 01:50:27.292454 23122 ts_manager.cc:194] Re-registered known tserver with Master: 54bb296f47a247b5b75d37f7f4c5117d (127.20.1.129:45333)
I20260620 01:50:27.293978 23194 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:48251
I20260620 01:50:27.295238 23926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:27.296025 23743 ts_manager.cc:194] Registered new tserver with Master: 54bb296f47a247b5b75d37f7f4c5117d (127.20.1.129:45333)
I20260620 01:50:27.298617 23727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5043f03c94574e408482f1e346500ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:27.299099 23727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:27.409783 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 23802
I20260620 01:50:27.430824 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:45333
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--webserver_port=40657
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
W20260620 01:50:27.889426 23946 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:27.889812 23946 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:27.889953 23946 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:50:27.890044 23946 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:27.900488 23946 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:27.900830 23946 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:27.915230 23946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:45333
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=40657
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:27.918639 23946 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:27.921597 23946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:27.937925 23952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:27.938146 23951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:27.938215 23954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:27.941309 23946 server_base.cc:1061] running on GCE node
I20260620 01:50:27.942315 23946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:27.944331 23946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:27.945788 23946 hybrid_clock.cc:648] HybridClock initialized: now 1781920227945685 us; error 56 us; skew 500 ppm
I20260620 01:50:27.946411 23946 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:27.949642 23946 webserver.cc:533] Webserver started at http://127.20.1.129:40657/ using document root <none> and password file <none>
I20260620 01:50:27.950760 23946 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:27.950973 23946 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:27.958774 23946 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:50:27.964524 23960 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:27.966753 23946 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260620 01:50:27.967017 23946 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "54bb296f47a247b5b75d37f7f4c5117d"
format_stamp: "Formatted at 2026-06-20 01:50:24 on dist-test-slave-wg14"
I20260620 01:50:27.968199 23946 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:27.995885 23946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:27.997025 23946 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:27.997629 23946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:27.999308 23946 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:28.002023 23946 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:28.002172 23946 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:28.002287 23946 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:28.002362 23946 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:50:28.002820 23946 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:50:28.002910 23946 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:50:28.002986 23946 fault_injection.cc:59] injecting fault for kudu (pid 23946): FLAGS_fault_before_start (process will exit)
I20260620 01:50:28.018726 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:45333
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--webserver_port=40657
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260620 01:50:28.478753 23968 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:28.479360 23968 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:28.479516 23968 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:28.489982 23968 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:28.490252 23968 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:28.505935 23968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:45333
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=40657
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:28.508119 23968 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:28.510464 23968 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:28.523572 23976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:28.523464 23975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:28.524112 23978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:28.525074 23968 server_base.cc:1061] running on GCE node
I20260620 01:50:28.525880 23968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:28.527483 23968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:28.528988 23968 hybrid_clock.cc:648] HybridClock initialized: now 1781920228528898 us; error 58 us; skew 500 ppm
I20260620 01:50:28.529577 23968 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:28.533010 23968 webserver.cc:533] Webserver started at http://127.20.1.129:40657/ using document root <none> and password file <none>
I20260620 01:50:28.534201 23968 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:28.534339 23968 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:28.541625 23968 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:50:28.545821 23984 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:28.547924 23968 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 01:50:28.548193 23968 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "54bb296f47a247b5b75d37f7f4c5117d"
format_stamp: "Formatted at 2026-06-20 01:50:24 on dist-test-slave-wg14"
I20260620 01:50:28.549036 23968 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:28.587117 23968 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:28.588339 23968 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:28.588681 23968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:28.589970 23968 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:28.592901 23968 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:28.593070 23968 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:28.593243 23968 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:28.593313 23968 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:28.638622 23968 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:45333
I20260620 01:50:28.638674 24096 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:45333 every 8 connection(s)
I20260620 01:50:28.640827 23968 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:28.649257 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 23968
I20260620 01:50:28.649765 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 23968
I20260620 01:50:28.681869 24098 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44517
I20260620 01:50:28.681984 24099 heartbeater.cc:344] Connected to a master server at 127.20.1.189:45519
I20260620 01:50:28.682430 24099 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:28.681900 24097 heartbeater.cc:344] Connected to a master server at 127.20.1.188:34077
I20260620 01:50:28.682435 24098 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:28.682686 24097 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:28.684490 24097 heartbeater.cc:507] Master 127.20.1.188:34077 requested a full tablet report, sending...
I20260620 01:50:28.684559 24099 heartbeater.cc:507] Master 127.20.1.189:45519 requested a full tablet report, sending...
I20260620 01:50:28.684573 24098 heartbeater.cc:507] Master 127.20.1.190:44517 requested a full tablet report, sending...
I20260620 01:50:28.687073 23194 ts_manager.cc:194] Re-registered known tserver with Master: 54bb296f47a247b5b75d37f7f4c5117d (127.20.1.129:45333)
I20260620 01:50:28.687191 23743 ts_manager.cc:194] Re-registered known tserver with Master: 54bb296f47a247b5b75d37f7f4c5117d (127.20.1.129:45333)
I20260620 01:50:28.688177 23194 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:49307
I20260620 01:50:28.689285 23122 ts_manager.cc:194] Re-registered known tserver with Master: 54bb296f47a247b5b75d37f7f4c5117d (127.20.1.129:45333)
I20260620 01:50:28.693728 24092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:28.793767 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 23968
I20260620 01:50:28.810957 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 23429
I20260620 01:50:28.823283 23551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:28.925697 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 23429
I20260620 01:50:28.941545 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 23566
I20260620 01:50:28.951844 23688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:29.062816 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 23566
I20260620 01:50:29.079442 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 23706
I20260620 01:50:29.087746 23774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:29.212797 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 23706
I20260620 01:50:29.234301 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 23088
I20260620 01:50:29.235729 23152 generic_service.cc:196] Checking for leaks (request via RPC)
W20260620 01:50:29.235791 23180 consensus_peers.cc:597] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 -> Peer 94716efbcd394c87b6b422791fce713d (127.20.1.190:44517): Couldn't send request to peer 94716efbcd394c87b6b422791fce713d. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:44517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 01:50:29.361598 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 23088
I20260620 01:50:29.382493 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 23159
I20260620 01:50:29.384546 23224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:29.518463 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 23159
2026-06-20T01:50:29Z chronyd exiting
2026-06-20T01:50:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:50:29Z Disabled control of system clock
I20260620 01:50:29.581390 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44517
--webserver_interface=127.20.1.190
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077 with env {}
W20260620 01:50:30.028198 24122 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:30.028643 24122 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:30.028735 24122 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:30.041509 24122 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:30.041644 24122 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:30.041703 24122 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:30.041745 24122 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:30.054826 24122 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:30.056829 24122 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:30.059649 24122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:30.073066 24128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:30.073096 24127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:30.075560 24130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:30.076434 24122 server_base.cc:1061] running on GCE node
I20260620 01:50:30.077728 24122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:30.079934 24122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:30.081646 24122 hybrid_clock.cc:648] HybridClock initialized: now 1781920230081483 us; error 142 us; skew 500 ppm
I20260620 01:50:30.082437 24122 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:30.089255 24122 webserver.cc:533] Webserver started at http://127.20.1.190:43201/ using document root <none> and password file <none>
I20260620 01:50:30.090960 24122 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:30.091286 24122 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:30.100836 24122 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:50:30.109997 24136 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:30.112947 24122 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:50:30.113257 24122 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "94716efbcd394c87b6b422791fce713d"
format_stamp: "Formatted at 2026-06-20 01:50:21 on dist-test-slave-wg14"
I20260620 01:50:30.114262 24122 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:30.144564 24122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:30.145717 24122 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:30.146148 24122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:30.175984 24187 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44517 every 8 connection(s)
I20260620 01:50:30.175979 24122 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44517
I20260620 01:50:30.178025 24122 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:50:30.178835 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 24122
I20260620 01:50:30.179070 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:50:30.180332 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:45519
--webserver_interface=127.20.1.189
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077 with env {}
I20260620 01:50:30.190459 24188 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:50:30.195325 24188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: Bootstrap starting.
I20260620 01:50:30.228765 24188 log.cc:826] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:30.248734 24188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:50:30.249449 24188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: Bootstrap complete.
I20260620 01:50:30.257984 24188 raft_consensus.cc:359] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:30.259097 24188 raft_consensus.cc:740] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94716efbcd394c87b6b422791fce713d, State: Initialized, Role: FOLLOWER
I20260620 01:50:30.261014 24188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:30.264256 24192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:30.264757 24192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:30.271185 24188 sys_catalog.cc:565] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:30.296505 24203 catalog_manager.cc:1294] Loaded cluster ID: 588c15584dca4915a1945314a0077392
I20260620 01:50:30.296875 24203 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: loading cluster ID for follower catalog manager: success
I20260620 01:50:30.306031 24203 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: acquiring CA information for follower catalog manager: success
I20260620 01:50:30.310340 24203 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:50:30.335958 24122 master_runner.cc:187] Error getting master registration for 127.20.1.189:45519: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45519: connect: Connection refused (error 111),
I20260620 01:50:30.342584 24122 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:50:30.671983 24190 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:30.672425 24190 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:30.672583 24190 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:30.682938 24190 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:30.683092 24190 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:30.683494 24190 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:30.683678 24190 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:30.696595 24190 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:45519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:30.698419 24190 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:30.700766 24190 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:30.714074 24212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:30.714102 24211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:30.715435 24214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:30.717530 24190 server_base.cc:1061] running on GCE node
I20260620 01:50:30.719089 24190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:30.721288 24190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:30.722826 24190 hybrid_clock.cc:648] HybridClock initialized: now 1781920230722748 us; error 85 us; skew 500 ppm
I20260620 01:50:30.723335 24190 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:30.726374 24190 webserver.cc:533] Webserver started at http://127.20.1.189:43303/ using document root <none> and password file <none>
I20260620 01:50:30.727428 24190 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:30.727596 24190 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:30.735872 24190 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.002s
I20260620 01:50:30.740643 24220 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:30.742774 24190 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260620 01:50:30.743045 24190 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "2d65d5070b2347bf8f6f12e6b1865c9a"
format_stamp: "Formatted at 2026-06-20 01:50:22 on dist-test-slave-wg14"
I20260620 01:50:30.744143 24190 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:30.780447 24190 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:30.781921 24190 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:30.782328 24190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:30.811880 24190 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:45519
I20260620 01:50:30.811880 24271 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:45519 every 8 connection(s)
I20260620 01:50:30.814105 24190 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:50:30.818286 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 24190
I20260620 01:50:30.818548 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:50:30.819818 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:34077
--webserver_interface=127.20.1.188
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077 with env {}
I20260620 01:50:30.824968 24272 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:50:30.829505 24272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a: Bootstrap starting.
I20260620 01:50:30.850695 24272 log.cc:826] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:30.862850 24272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:50:30.863592 24272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a: Bootstrap complete.
I20260620 01:50:30.871660 24272 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:30.872601 24272 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d65d5070b2347bf8f6f12e6b1865c9a, State: Initialized, Role: FOLLOWER
I20260620 01:50:30.873644 24272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:30.874729 24276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:30.875080 24276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:30.876012 24272 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:30.899210 24287 catalog_manager.cc:1294] Loaded cluster ID: 588c15584dca4915a1945314a0077392
I20260620 01:50:30.899377 24287 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a: loading cluster ID for follower catalog manager: success
I20260620 01:50:30.904584 24287 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a: acquiring CA information for follower catalog manager: success
I20260620 01:50:30.909312 24287 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:50:30.939898 24190 master_runner.cc:187] Error getting master registration for 127.20.1.188:34077: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34077: connect: Connection refused (error 111),
I20260620 01:50:30.945679 24190 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:50:31.353494 24274 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:31.353992 24274 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:31.354154 24274 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:31.364609 24274 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:31.364758 24274 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:31.364845 24274 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:31.364917 24274 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:31.379920 24274 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:34077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:31.382261 24274 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:31.384948 24274 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:31.399348 24296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:31.399350 24297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:31.400593 24299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:31.400907 24274 server_base.cc:1061] running on GCE node
I20260620 01:50:31.402184 24274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:31.404402 24274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:31.405725 24274 hybrid_clock.cc:648] HybridClock initialized: now 1781920231405654 us; error 77 us; skew 500 ppm
I20260620 01:50:31.406209 24274 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:31.409299 24274 webserver.cc:533] Webserver started at http://127.20.1.188:39003/ using document root <none> and password file <none>
I20260620 01:50:31.410311 24274 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:31.410490 24274 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:31.417971 24274 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:50:31.422258 24305 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:31.424872 24274 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:50:31.425602 24274 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "5043f03c94574e408482f1e346500ae2"
format_stamp: "Formatted at 2026-06-20 01:50:22 on dist-test-slave-wg14"
I20260620 01:50:31.426911 24274 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:31.453431 24274 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:31.454602 24274 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:31.454993 24274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:31.482036 24274 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:34077
I20260620 01:50:31.482051 24356 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:34077 every 8 connection(s)
I20260620 01:50:31.484030 24274 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:50:31.492547 24357 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:50:31.494769 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 24274
I20260620 01:50:31.495046 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:50:31.497126 24357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2: Bootstrap starting.
I20260620 01:50:31.523602 24357 log.cc:826] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:31.538089 24357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:50:31.538738 24357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2: Bootstrap complete.
I20260620 01:50:31.547209 24357 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:31.548110 24357 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5043f03c94574e408482f1e346500ae2, State: Initialized, Role: FOLLOWER
I20260620 01:50:31.549028 24357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:31.550885 24364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:31.551555 24364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:31.553015 24357 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:31.571225 24375 catalog_manager.cc:1294] Loaded cluster ID: 588c15584dca4915a1945314a0077392
I20260620 01:50:31.571376 24375 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2: loading cluster ID for follower catalog manager: success
I20260620 01:50:31.576472 24375 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2: acquiring CA information for follower catalog manager: success
I20260620 01:50:31.580470 24375 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:50:31.613965 24274 master_runner.cc:264] No leader master found from master 5043f03c94574e408482f1e346500ae2
I20260620 01:50:31.614215 24274 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:50:31.678194 24378 raft_consensus.cc:493] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:50:31.678540 24378 raft_consensus.cc:515] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:31.681697 24378 leader_election.cc:290] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2d65d5070b2347bf8f6f12e6b1865c9a (127.20.1.189:45519), 5043f03c94574e408482f1e346500ae2 (127.20.1.188:34077)
I20260620 01:50:31.692785 24247 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94716efbcd394c87b6b422791fce713d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" is_pre_election: true
I20260620 01:50:31.693511 24247 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94716efbcd394c87b6b422791fce713d in term 1.
I20260620 01:50:31.694404 24138 leader_election.cc:304] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d65d5070b2347bf8f6f12e6b1865c9a, 94716efbcd394c87b6b422791fce713d; no voters:
I20260620 01:50:31.695475 24332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94716efbcd394c87b6b422791fce713d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5043f03c94574e408482f1e346500ae2" is_pre_election: true
I20260620 01:50:31.696410 24378 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:50:31.696671 24378 raft_consensus.cc:493] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:50:31.696775 24378 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:50:31.696698 24332 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94716efbcd394c87b6b422791fce713d in term 1.
I20260620 01:50:31.701701 24378 raft_consensus.cc:515] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:31.702612 24378 leader_election.cc:290] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [CANDIDATE]: Term 2 election: Requested vote from peers 2d65d5070b2347bf8f6f12e6b1865c9a (127.20.1.189:45519), 5043f03c94574e408482f1e346500ae2 (127.20.1.188:34077)
I20260620 01:50:31.703609 24247 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94716efbcd394c87b6b422791fce713d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a"
I20260620 01:50:31.703926 24247 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:50:31.703900 24332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94716efbcd394c87b6b422791fce713d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5043f03c94574e408482f1e346500ae2"
I20260620 01:50:31.704181 24332 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:50:31.709440 24247 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94716efbcd394c87b6b422791fce713d in term 2.
I20260620 01:50:31.710050 24138 leader_election.cc:304] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d65d5070b2347bf8f6f12e6b1865c9a, 94716efbcd394c87b6b422791fce713d; no voters:
I20260620 01:50:31.710412 24378 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 01:50:31.710636 24332 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94716efbcd394c87b6b422791fce713d in term 2.
I20260620 01:50:31.711169 24378 raft_consensus.cc:697] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 2 LEADER]: Becoming Leader. State: Replica: 94716efbcd394c87b6b422791fce713d, State: Running, Role: LEADER
I20260620 01:50:31.711997 24378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:31.716588 24381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94716efbcd394c87b6b422791fce713d. Latest consensus state: current_term: 2 leader_uuid: "94716efbcd394c87b6b422791fce713d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:31.716975 24381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:31.719416 24383 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:50:31.725289 24383 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:50:31.727679 24383 catalog_manager.cc:1294] Loaded cluster ID: 588c15584dca4915a1945314a0077392
I20260620 01:50:31.728003 24383 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:50:31.730266 24383 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:50:31.731978 24383 catalog_manager.cc:6080] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: Loaded TSK: 0
I20260620 01:50:31.734054 24383 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:50:31.764056 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--builtin_ntp_servers=127.20.1.148:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:50:32.095901 24247 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 2 FOLLOWER]: Refusing update from remote peer 94716efbcd394c87b6b422791fce713d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:50:32.097813 24381 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:50:32.104013 24385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94716efbcd394c87b6b422791fce713d. Latest consensus state: current_term: 2 leader_uuid: "94716efbcd394c87b6b422791fce713d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:32.104403 24385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:32.106724 24378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "94716efbcd394c87b6b422791fce713d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:32.107088 24378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:32.108336 24332 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 2 FOLLOWER]: Refusing update from remote peer 94716efbcd394c87b6b422791fce713d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:50:32.109041 24385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "94716efbcd394c87b6b422791fce713d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:32.109468 24385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:32.109689 24378 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:50:32.115903 24392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "94716efbcd394c87b6b422791fce713d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:32.116489 24392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:32.117157 24389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94716efbcd394c87b6b422791fce713d. Latest consensus state: current_term: 2 leader_uuid: "94716efbcd394c87b6b422791fce713d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:32.117578 24389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:50:32.263264 24384 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:32.263639 24384 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:32.263742 24384 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:32.273378 24384 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:32.273619 24384 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:32.286276 24384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:32.288120 24384 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:32.290098 24384 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:32.304929 24398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:32.305387 24397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:32.306361 24400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:32.308204 24384 server_base.cc:1061] running on GCE node
I20260620 01:50:32.309096 24384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:32.310575 24384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:32.311890 24384 hybrid_clock.cc:648] HybridClock initialized: now 1781920232311808 us; error 91 us; skew 500 ppm
I20260620 01:50:32.312309 24384 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:32.317153 24384 webserver.cc:533] Webserver started at http://127.20.1.129:41311/ using document root <none> and password file <none>
I20260620 01:50:32.318549 24384 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:32.318719 24384 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:32.328022 24384 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:50:32.334137 24406 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:32.337000 24384 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260620 01:50:32.337285 24384 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "54bb296f47a247b5b75d37f7f4c5117d"
format_stamp: "Formatted at 2026-06-20 01:50:24 on dist-test-slave-wg14"
I20260620 01:50:32.338289 24384 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:32.360567 24384 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:32.362010 24384 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:32.362421 24384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:32.364239 24384 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:32.366572 24384 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:32.366690 24384 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:32.366797 24384 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:32.366833 24384 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:32.420732 24384 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38929
I20260620 01:50:32.420856 24518 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38929 every 8 connection(s)
I20260620 01:50:32.422828 24384 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:32.426168 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 24384
I20260620 01:50:32.426404 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:50:32.443048 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--builtin_ntp_servers=127.20.1.148:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:50:32.456836 24521 heartbeater.cc:344] Connected to a master server at 127.20.1.189:45519
I20260620 01:50:32.457033 24520 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44517
I20260620 01:50:32.457311 24520 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:32.457307 24521 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:32.458577 24521 heartbeater.cc:507] Master 127.20.1.189:45519 requested a full tablet report, sending...
I20260620 01:50:32.458638 24520 heartbeater.cc:507] Master 127.20.1.190:44517 requested a full tablet report, sending...
I20260620 01:50:32.462494 24237 ts_manager.cc:194] Registered new tserver with Master: 54bb296f47a247b5b75d37f7f4c5117d (127.20.1.129:38929)
I20260620 01:50:32.470824 24153 ts_manager.cc:194] Registered new tserver with Master: 54bb296f47a247b5b75d37f7f4c5117d (127.20.1.129:38929)
I20260620 01:50:32.473332 24519 heartbeater.cc:344] Connected to a master server at 127.20.1.188:34077
I20260620 01:50:32.473630 24519 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:32.474323 24519 heartbeater.cc:507] Master 127.20.1.188:34077 requested a full tablet report, sending...
I20260620 01:50:32.474236 24153 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:43063
I20260620 01:50:32.477274 24322 ts_manager.cc:194] Registered new tserver with Master: 54bb296f47a247b5b75d37f7f4c5117d (127.20.1.129:38929)
I20260620 01:50:32.600041 24378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "94716efbcd394c87b6b422791fce713d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:32.600396 24378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: This master's current role is: LEADER
W20260620 01:50:32.909150 24529 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:32.909554 24529 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:32.909669 24529 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:32.919296 24529 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:32.919739 24529 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:50:32.932739 24529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:32.934702 24529 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:32.937067 24529 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:32.951284 24535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:32.951614 24536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:32.953812 24538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:32.954342 24529 server_base.cc:1061] running on GCE node
I20260620 01:50:32.955304 24529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:32.956887 24529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:32.960450 24529 hybrid_clock.cc:648] HybridClock initialized: now 1781920232960336 us; error 166 us; skew 500 ppm
I20260620 01:50:32.962050 24529 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:32.967959 24529 webserver.cc:533] Webserver started at http://127.20.1.130:41947/ using document root <none> and password file <none>
I20260620 01:50:32.968957 24529 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:32.969091 24529 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:32.977659 24529 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.003s
I20260620 01:50:32.982551 24544 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:32.985118 24529 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260620 01:50:32.985425 24529 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "e52eb4213cfc4dc6aaa391a9ec00de90"
format_stamp: "Formatted at 2026-06-20 01:50:24 on dist-test-slave-wg14"
I20260620 01:50:32.991881 24529 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:33.028482 24529 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:33.029574 24529 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:33.029922 24529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:33.031476 24529 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:33.034974 24529 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:33.035148 24529 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:33.035269 24529 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:33.035306 24529 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:33.088043 24529 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:37383
I20260620 01:50:33.088032 24656 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:37383 every 8 connection(s)
I20260620 01:50:33.090080 24529 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:50:33.095400 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 24529
I20260620 01:50:33.095630 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
I20260620 01:50:33.105625 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--builtin_ntp_servers=127.20.1.148:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:50:33.134151 24659 heartbeater.cc:344] Connected to a master server at 127.20.1.189:45519
I20260620 01:50:33.134673 24659 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:33.136018 24659 heartbeater.cc:507] Master 127.20.1.189:45519 requested a full tablet report, sending...
I20260620 01:50:33.137001 24658 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44517
I20260620 01:50:33.137202 24658 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:33.137825 24658 heartbeater.cc:507] Master 127.20.1.190:44517 requested a full tablet report, sending...
I20260620 01:50:33.138161 24657 heartbeater.cc:344] Connected to a master server at 127.20.1.188:34077
I20260620 01:50:33.138324 24657 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:33.138847 24657 heartbeater.cc:507] Master 127.20.1.188:34077 requested a full tablet report, sending...
I20260620 01:50:33.139734 24153 ts_manager.cc:194] Registered new tserver with Master: e52eb4213cfc4dc6aaa391a9ec00de90 (127.20.1.130:37383)
I20260620 01:50:33.139860 24237 ts_manager.cc:194] Registered new tserver with Master: e52eb4213cfc4dc6aaa391a9ec00de90 (127.20.1.130:37383)
I20260620 01:50:33.140479 24322 ts_manager.cc:194] Registered new tserver with Master: e52eb4213cfc4dc6aaa391a9ec00de90 (127.20.1.130:37383)
I20260620 01:50:33.141280 24153 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.130:56385
I20260620 01:50:33.479792 24520 heartbeater.cc:499] Master 127.20.1.190:44517 was elected leader, sending a full tablet report...
W20260620 01:50:33.560993 24665 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:33.561421 24665 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:33.561553 24665 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:33.572454 24665 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:33.572641 24665 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:50:33.584939 24665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:33.587316 24665 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:33.589856 24665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:33.604959 24672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:33.605162 24675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:33.604985 24673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:33.606173 24665 server_base.cc:1061] running on GCE node
I20260620 01:50:33.606978 24665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:33.608414 24665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:33.609957 24665 hybrid_clock.cc:648] HybridClock initialized: now 1781920233609846 us; error 83 us; skew 500 ppm
I20260620 01:50:33.610476 24665 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:33.614631 24665 webserver.cc:533] Webserver started at http://127.20.1.131:36129/ using document root <none> and password file <none>
I20260620 01:50:33.615777 24665 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:33.616215 24665 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:33.624372 24665 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 01:50:33.628650 24681 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:33.630542 24665 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 01:50:33.630801 24665 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "78bdd4be122c41ad9cd7f732127cb11d"
format_stamp: "Formatted at 2026-06-20 01:50:25 on dist-test-slave-wg14"
I20260620 01:50:33.631771 24665 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:33.661895 24665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:33.663318 24665 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:33.663671 24665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:33.664942 24665 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:33.667093 24665 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:33.667225 24665 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 01:50:33.667380 24665 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:33.667477 24665 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:33.710417 24665 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:33215
I20260620 01:50:33.710461 24793 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:33215 every 8 connection(s)
I20260620 01:50:33.712961 24665 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:50:33.720075 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 24665
I20260620 01:50:33.720340 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
I20260620 01:50:33.741523 24795 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44517
I20260620 01:50:33.741559 24796 heartbeater.cc:344] Connected to a master server at 127.20.1.189:45519
I20260620 01:50:33.741587 24794 heartbeater.cc:344] Connected to a master server at 127.20.1.188:34077
I20260620 01:50:33.742103 24795 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:33.742105 24796 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:33.742105 24794 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:33.743891 24794 heartbeater.cc:507] Master 127.20.1.188:34077 requested a full tablet report, sending...
I20260620 01:50:33.743857 24795 heartbeater.cc:507] Master 127.20.1.190:44517 requested a full tablet report, sending...
I20260620 01:50:33.743924 24796 heartbeater.cc:507] Master 127.20.1.189:45519 requested a full tablet report, sending...
I20260620 01:50:33.746508 24153 ts_manager.cc:194] Registered new tserver with Master: 78bdd4be122c41ad9cd7f732127cb11d (127.20.1.131:33215)
I20260620 01:50:33.746585 24237 ts_manager.cc:194] Registered new tserver with Master: 78bdd4be122c41ad9cd7f732127cb11d (127.20.1.131:33215)
I20260620 01:50:33.746591 24322 ts_manager.cc:194] Registered new tserver with Master: 78bdd4be122c41ad9cd7f732127cb11d (127.20.1.131:33215)
I20260620 01:50:33.747793 24153 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.131:55807
I20260620 01:50:33.757717 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:50:33.771924 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:50:33.772403 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 24122
I20260620 01:50:33.774205 24183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:33.904533 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 24122
W20260620 01:50:33.925051 24307 connection.cc:573] server connection from 127.0.0.1:53086 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:50:33.925810 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44517
--webserver_interface=127.20.1.190
--webserver_port=43201
--builtin_ntp_servers=127.20.1.148:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077 with env {}
I20260620 01:50:34.144270 24658 heartbeater.cc:499] Master 127.20.1.190:44517 was elected leader, sending a full tablet report...
W20260620 01:50:34.146361 24658 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:44517 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:44517: connect: Connection refused (error 111)
W20260620 01:50:34.389117 24805 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:34.389580 24805 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:34.389716 24805 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:34.404310 24805 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:34.404462 24805 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:34.404553 24805 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:34.404616 24805 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:34.417879 24805 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=43201
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:34.420035 24805 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:34.422506 24805 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:34.435549 24812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:34.436383 24811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:34.436825 24814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:34.442883 24805 server_base.cc:1061] running on GCE node
I20260620 01:50:34.444006 24805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:34.447367 24805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:34.448686 24805 hybrid_clock.cc:648] HybridClock initialized: now 1781920234448639 us; error 44 us; skew 500 ppm
I20260620 01:50:34.449155 24805 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:34.454043 24805 webserver.cc:533] Webserver started at http://127.20.1.190:43201/ using document root <none> and password file <none>
I20260620 01:50:34.456168 24805 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:34.456722 24805 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:34.465813 24805 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.000s sys 0.004s
I20260620 01:50:34.472537 24820 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:34.475373 24805 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.003s sys 0.003s
I20260620 01:50:34.475639 24805 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "94716efbcd394c87b6b422791fce713d"
format_stamp: "Formatted at 2026-06-20 01:50:21 on dist-test-slave-wg14"
I20260620 01:50:34.476603 24805 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260620 01:50:34.500648 24520 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:44517 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:44517: connect: Connection refused (error 111)
I20260620 01:50:34.518250 24805 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:34.519397 24805 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:34.519760 24805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:34.552893 24805 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44517
I20260620 01:50:34.552893 24872 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44517 every 8 connection(s)
I20260620 01:50:34.555188 24805 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:50:34.556529 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 24805
I20260620 01:50:34.556905 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 24384
I20260620 01:50:34.567231 24873 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:50:34.571962 24514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:34.572238 24873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: Bootstrap starting.
I20260620 01:50:34.601630 24873 log.cc:826] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:34.619222 24873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:50:34.620052 24873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: Bootstrap complete.
I20260620 01:50:34.633716 24873 raft_consensus.cc:359] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:34.635939 24873 raft_consensus.cc:740] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94716efbcd394c87b6b422791fce713d, State: Initialized, Role: FOLLOWER
I20260620 01:50:34.637240 24873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:34.638458 24879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:34.638813 24879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:34.640661 24873 sys_catalog.cc:565] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:34.665241 24890 catalog_manager.cc:1294] Loaded cluster ID: 588c15584dca4915a1945314a0077392
I20260620 01:50:34.665782 24890 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: loading cluster ID for follower catalog manager: success
I20260620 01:50:34.673602 24890 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: acquiring CA information for follower catalog manager: success
I20260620 01:50:34.677678 24890 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:50:34.715951 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 24384
I20260620 01:50:34.736414 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:38929
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--webserver_port=41311
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:50:34.752555 24795 heartbeater.cc:499] Master 127.20.1.190:44517 was elected leader, sending a full tablet report...
I20260620 01:50:34.758723 24805 master_runner.cc:264] No leader master found from master 94716efbcd394c87b6b422791fce713d
I20260620 01:50:34.758920 24805 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:50:34.765715 24838 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "78bdd4be122c41ad9cd7f732127cb11d" instance_seqno: 1781920233699733) as {username='slave'} at 127.20.1.131:52797; Asking this server to re-register.
I20260620 01:50:34.767057 24795 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:34.769850 24838 ts_manager.cc:194] Registered new tserver with Master: 78bdd4be122c41ad9cd7f732127cb11d (127.20.1.131:33215)
I20260620 01:50:35.127728 24898 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 94716efbcd394c87b6b422791fce713d)
I20260620 01:50:35.128513 24898 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:35.138996 24898 leader_election.cc:290] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 94716efbcd394c87b6b422791fce713d (127.20.1.190:44517), 5043f03c94574e408482f1e346500ae2 (127.20.1.188:34077)
I20260620 01:50:35.177963 24658 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44517
I20260620 01:50:35.178262 24658 heartbeater.cc:499] Master 127.20.1.190:44517 was elected leader, sending a full tablet report...
I20260620 01:50:35.179746 24838 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e52eb4213cfc4dc6aaa391a9ec00de90" instance_seqno: 1781920233076201) as {username='slave'} at 127.20.1.130:47507; Asking this server to re-register.
I20260620 01:50:35.180789 24658 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:35.180815 24848 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "94716efbcd394c87b6b422791fce713d" is_pre_election: true
I20260620 01:50:35.181478 24848 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d65d5070b2347bf8f6f12e6b1865c9a in term 2.
I20260620 01:50:35.182384 24223 leader_election.cc:304] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d65d5070b2347bf8f6f12e6b1865c9a, 94716efbcd394c87b6b422791fce713d; no voters:
I20260620 01:50:35.182971 24898 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 01:50:35.183096 24898 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 94716efbcd394c87b6b422791fce713d)
I20260620 01:50:35.183214 24898 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:50:35.187801 24838 ts_manager.cc:194] Registered new tserver with Master: e52eb4213cfc4dc6aaa391a9ec00de90 (127.20.1.130:37383)
I20260620 01:50:35.189510 24898 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:35.191308 24848 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "94716efbcd394c87b6b422791fce713d"
I20260620 01:50:35.191632 24848 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:50:35.191648 24332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5043f03c94574e408482f1e346500ae2" is_pre_election: true
I20260620 01:50:35.194265 24332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5043f03c94574e408482f1e346500ae2"
I20260620 01:50:35.194800 24898 leader_election.cc:290] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [CANDIDATE]: Term 3 election: Requested vote from peers 94716efbcd394c87b6b422791fce713d (127.20.1.190:44517), 5043f03c94574e408482f1e346500ae2 (127.20.1.188:34077)
I20260620 01:50:35.199702 24848 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d65d5070b2347bf8f6f12e6b1865c9a in term 3.
I20260620 01:50:35.200495 24223 leader_election.cc:304] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d65d5070b2347bf8f6f12e6b1865c9a, 94716efbcd394c87b6b422791fce713d; no voters: 5043f03c94574e408482f1e346500ae2
I20260620 01:50:35.201305 24898 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 3 FOLLOWER]: Leader election won for term 3
I20260620 01:50:35.211794 24898 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [term 3 LEADER]: Becoming Leader. State: Replica: 2d65d5070b2347bf8f6f12e6b1865c9a, State: Running, Role: LEADER
I20260620 01:50:35.213523 24898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:35.222630 24903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d65d5070b2347bf8f6f12e6b1865c9a. Latest consensus state: current_term: 3 leader_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:35.223295 24903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:35.230127 24908 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:50:35.233050 24908 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:50:35.234346 24908 catalog_manager.cc:1294] Loaded cluster ID: 588c15584dca4915a1945314a0077392
I20260620 01:50:35.234462 24908 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:50:35.236230 24908 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:50:35.237661 24908 catalog_manager.cc:6080] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a: Loaded TSK: 0
I20260620 01:50:35.239324 24908 catalog_manager.cc:1549] Initializing in-progress tserver states...
W20260620 01:50:35.242081 24896 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:35.242552 24896 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:35.242769 24896 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:35.253830 24896 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:35.254555 24896 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:35.267863 24896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:38929
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=41311
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:35.269807 24896 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:35.272193 24896 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:35.289535 24910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:35.289881 24911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:35.289986 24913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:35.291018 24896 server_base.cc:1061] running on GCE node
I20260620 01:50:35.292645 24896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:35.294059 24896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:35.297710 24896 hybrid_clock.cc:648] HybridClock initialized: now 1781920235297507 us; error 145 us; skew 500 ppm
I20260620 01:50:35.298280 24896 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:35.302536 24896 webserver.cc:533] Webserver started at http://127.20.1.129:41311/ using document root <none> and password file <none>
I20260620 01:50:35.303522 24896 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:35.303651 24896 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:35.311604 24896 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 01:50:35.316808 24919 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:35.319705 24896 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:50:35.320202 24896 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "54bb296f47a247b5b75d37f7f4c5117d"
format_stamp: "Formatted at 2026-06-20 01:50:24 on dist-test-slave-wg14"
I20260620 01:50:35.321707 24896 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:35.342388 24896 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:35.343636 24896 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:35.344002 24896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:35.345221 24896 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:35.347410 24896 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:35.347517 24896 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:35.347649 24896 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:35.347705 24896 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:35.401788 24896 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38929
I20260620 01:50:35.401834 25031 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38929 every 8 connection(s)
I20260620 01:50:35.403900 24896 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:35.413667 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 24896
I20260620 01:50:35.414021 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 24896
I20260620 01:50:35.420181 25035 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 94716efbcd394c87b6b422791fce713d)
I20260620 01:50:35.421087 25035 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } }
I20260620 01:50:35.437775 24247 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5043f03c94574e408482f1e346500ae2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" is_pre_election: true
I20260620 01:50:35.439683 25035 leader_election.cc:290] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 94716efbcd394c87b6b422791fce713d (127.20.1.190:44517), 2d65d5070b2347bf8f6f12e6b1865c9a (127.20.1.189:45519)
I20260620 01:50:35.444844 25032 heartbeater.cc:344] Connected to a master server at 127.20.1.188:34077
I20260620 01:50:35.445487 25032 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:35.446949 25034 heartbeater.cc:344] Connected to a master server at 127.20.1.189:45519
I20260620 01:50:35.447115 25032 heartbeater.cc:507] Master 127.20.1.188:34077 requested a full tablet report, sending...
I20260620 01:50:35.447206 25034 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:35.449024 25034 heartbeater.cc:507] Master 127.20.1.189:45519 requested a full tablet report, sending...
I20260620 01:50:35.451301 24321 ts_manager.cc:194] Re-registered known tserver with Master: 54bb296f47a247b5b75d37f7f4c5117d (127.20.1.129:38929)
I20260620 01:50:35.452382 25027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:35.452600 24237 ts_manager.cc:194] Re-registered known tserver with Master: 54bb296f47a247b5b75d37f7f4c5117d (127.20.1.129:38929)
I20260620 01:50:35.456146 24237 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:40835
I20260620 01:50:35.458299 24848 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5043f03c94574e408482f1e346500ae2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "94716efbcd394c87b6b422791fce713d" is_pre_election: true
I20260620 01:50:35.458818 24848 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5043f03c94574e408482f1e346500ae2 in current term 3: Already voted for candidate 2d65d5070b2347bf8f6f12e6b1865c9a in this term.
I20260620 01:50:35.459986 24308 leader_election.cc:304] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5043f03c94574e408482f1e346500ae2; no voters: 2d65d5070b2347bf8f6f12e6b1865c9a, 94716efbcd394c87b6b422791fce713d
I20260620 01:50:35.460892 25035 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:50:35.464665 25035 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 01:50:35.566910 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 24896
I20260620 01:50:35.583628 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:38929
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--webserver_port=41311
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
W20260620 01:50:35.583946 24893 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:50:35.414613 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:56949 (local address 127.20.1.190:44517)
0620 01:50:35.414854 (+ 241us) server_negotiation.cc:207] Beginning negotiation
0620 01:50:35.414863 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0620 01:50:35.441409 (+ 26546us) server_negotiation.cc:408] Connection header received
0620 01:50:35.445579 (+ 4170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:50:35.445585 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:50:35.445848 (+ 263us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:50:35.446501 (+ 653us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:50:35.448762 (+ 2261us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:50:35.450178 (+ 1416us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:50:35.452463 (+ 2285us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:50:35.453021 (+ 558us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0620 01:50:35.582944 (+129923us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:56949: BlockingRecv error: recv got EOF from 127.20.1.129:56949 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 01:50:35.761750 24332 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [term 3 FOLLOWER]: Refusing update from remote peer 2d65d5070b2347bf8f6f12e6b1865c9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:50:35.762625 25046 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:50:35.765110 25035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d65d5070b2347bf8f6f12e6b1865c9a. Latest consensus state: current_term: 3 leader_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:35.765476 25035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:35.767763 24848 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [term 3 FOLLOWER]: Refusing update from remote peer 2d65d5070b2347bf8f6f12e6b1865c9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:50:35.769230 25048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:35.769645 25048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:35.769351 25050 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 01:50:35.772881 25035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:35.773168 25035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5043f03c94574e408482f1e346500ae2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:35.778694 25057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:35.779428 25057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:35.780097 25054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d65d5070b2347bf8f6f12e6b1865c9a. Latest consensus state: current_term: 3 leader_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:35.780422 25054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94716efbcd394c87b6b422791fce713d [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:50:36.069391 25045 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:36.069844 25045 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:36.069995 25045 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:50:36.070070 25045 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:36.085304 25045 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:36.085572 25045 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:36.102795 25045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:38929
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=41311
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:36.104838 25045 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:36.107261 25045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:36.127306 25062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:36.129361 25045 server_base.cc:1061] running on GCE node
W20260620 01:50:36.129338 25065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:36.129347 25063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:36.130407 25045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:36.132001 25045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:36.136026 25045 hybrid_clock.cc:648] HybridClock initialized: now 1781920236135501 us; error 291 us; skew 500 ppm
I20260620 01:50:36.136586 25045 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:36.140233 25045 webserver.cc:533] Webserver started at http://127.20.1.129:41311/ using document root <none> and password file <none>
I20260620 01:50:36.141672 25045 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:36.141866 25045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:36.150141 25045 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.003s
I20260620 01:50:36.159338 25071 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:36.162140 25045 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260620 01:50:36.162389 25045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "54bb296f47a247b5b75d37f7f4c5117d"
format_stamp: "Formatted at 2026-06-20 01:50:24 on dist-test-slave-wg14"
I20260620 01:50:36.163609 25045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:36.201974 25045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:36.203608 25045 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:36.204015 25045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:36.205269 25045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:36.207417 25045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:36.207659 25045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:36.207769 25045 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:36.207829 25045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:50:36.208256 25045 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:50:36.208344 25045 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:50:36.208401 25045 fault_injection.cc:59] injecting fault for kudu (pid 25045): FLAGS_fault_before_start (process will exit)
I20260620 01:50:36.235247 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:38929
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--webserver_port=41311
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:50:36.327400 25050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94716efbcd394c87b6b422791fce713d" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44517 } } peers { permanent_uuid: "2d65d5070b2347bf8f6f12e6b1865c9a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45519 } } peers { permanent_uuid: "5043f03c94574e408482f1e346500ae2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34077 } } }
I20260620 01:50:36.327790 25050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d65d5070b2347bf8f6f12e6b1865c9a [sys.catalog]: This master's current role is: LEADER
W20260620 01:50:36.725718 25079 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:36.726438 25079 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:36.726671 25079 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:36.737442 25079 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:36.737661 25079 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:36.750146 25079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:38929
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=41311
--tserver_master_addrs=127.20.1.190:44517,127.20.1.189:45519,127.20.1.188:34077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:36.753621 25079 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:36.755837 24796 heartbeater.cc:499] Master 127.20.1.189:45519 was elected leader, sending a full tablet report...
I20260620 01:50:36.757505 25079 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:36.772089 25087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:36.772326 25088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:36.772768 25090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:36.774300 25079 server_base.cc:1061] running on GCE node
I20260620 01:50:36.775516 25079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:36.776767 25079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:36.778163 25079 hybrid_clock.cc:648] HybridClock initialized: now 1781920236778079 us; error 59 us; skew 500 ppm
I20260620 01:50:36.778625 25079 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:36.782192 25079 webserver.cc:533] Webserver started at http://127.20.1.129:41311/ using document root <none> and password file <none>
I20260620 01:50:36.783327 25079 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:36.783476 25079 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:36.791316 25079 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 01:50:36.797576 25096 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:36.800128 25079 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.004s
I20260620 01:50:36.800397 25079 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "54bb296f47a247b5b75d37f7f4c5117d"
format_stamp: "Formatted at 2026-06-20 01:50:24 on dist-test-slave-wg14"
I20260620 01:50:36.801281 25079 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:36.840309 25079 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:36.841395 25079 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:36.841759 25079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:36.843633 25079 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:36.846483 25079 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:36.846608 25079 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:36.846840 25079 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:36.846922 25079 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:36.896242 25079 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38929
I20260620 01:50:36.896312 25208 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38929 every 8 connection(s)
I20260620 01:50:36.898166 25079 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:36.908241 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 25079
I20260620 01:50:36.908614 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 25079
I20260620 01:50:36.946171 25211 heartbeater.cc:344] Connected to a master server at 127.20.1.189:45519
I20260620 01:50:36.946883 25211 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:36.947188 25210 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44517
I20260620 01:50:36.947408 25209 heartbeater.cc:344] Connected to a master server at 127.20.1.188:34077
I20260620 01:50:36.947671 25209 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:36.947815 25210 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:36.948801 25211 heartbeater.cc:507] Master 127.20.1.189:45519 requested a full tablet report, sending...
I20260620 01:50:36.950726 25210 heartbeater.cc:507] Master 127.20.1.190:44517 requested a full tablet report, sending...
I20260620 01:50:36.951433 25209 heartbeater.cc:507] Master 127.20.1.188:34077 requested a full tablet report, sending...
I20260620 01:50:36.951491 24237 ts_manager.cc:194] Re-registered known tserver with Master: 54bb296f47a247b5b75d37f7f4c5117d (127.20.1.129:38929)
I20260620 01:50:36.951704 25204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:36.952772 24237 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:50709
I20260620 01:50:36.953289 24838 ts_manager.cc:194] Registered new tserver with Master: 54bb296f47a247b5b75d37f7f4c5117d (127.20.1.129:38929)
I20260620 01:50:36.954172 24321 ts_manager.cc:194] Re-registered known tserver with Master: 54bb296f47a247b5b75d37f7f4c5117d (127.20.1.129:38929)
I20260620 01:50:37.072253 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 25079
I20260620 01:50:37.090067 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 24529
I20260620 01:50:37.103271 24652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:37.214041 24659 heartbeater.cc:499] Master 127.20.1.189:45519 was elected leader, sending a full tablet report...
I20260620 01:50:37.216638 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 24529
I20260620 01:50:37.233882 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 24665
I20260620 01:50:37.245335 24789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:37.358922 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 24665
I20260620 01:50:37.376137 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 24805
I20260620 01:50:37.386222 24868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:37.512831 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 24805
I20260620 01:50:37.532274 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 24190
I20260620 01:50:37.533903 24267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:37.660743 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 24190
I20260620 01:50:37.680702 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 24274
I20260620 01:50:37.682070 24352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:37.832145 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 24274
2026-06-20T01:50:37Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/1 (16856 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/2
2026-06-20T01:50:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:50:37Z Disabled control of system clock
2026-06-20T01:50:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:50:37Z Disabled control of system clock
2026-06-20T01:50:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:50:37Z Disabled control of system clock
2026-06-20T01:50:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:50:37Z Disabled control of system clock
2026-06-20T01:50:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:50:38Z Disabled control of system clock
I20260620 01:50:38.034828 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:35539
--webserver_interface=127.20.1.190
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217 with env {}
W20260620 01:50:38.524027 25240 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:38.524492 25240 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:38.524623 25240 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:38.535056 25240 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:38.535254 25240 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:38.535362 25240 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:38.535435 25240 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:38.548967 25240 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:35539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:38.551396 25240 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:38.555044 25240 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:38.568414 25248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:38.568313 25246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:38.568363 25245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:38.569769 25240 server_base.cc:1061] running on GCE node
I20260620 01:50:38.571556 25240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:38.573992 25240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:38.577525 25240 hybrid_clock.cc:648] HybridClock initialized: now 1781920238577263 us; error 165 us; skew 500 ppm
I20260620 01:50:38.578151 25240 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:38.581869 25240 webserver.cc:533] Webserver started at http://127.20.1.190:46771/ using document root <none> and password file <none>
I20260620 01:50:38.582895 25240 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:38.583292 25240 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:38.584061 25240 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:38.587770 25240 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "d7ebec69238c4564b5e9a92d34a663cf"
format_stamp: "Formatted at 2026-06-20 01:50:38 on dist-test-slave-wg14"
I20260620 01:50:38.588950 25240 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "d7ebec69238c4564b5e9a92d34a663cf"
format_stamp: "Formatted at 2026-06-20 01:50:38 on dist-test-slave-wg14"
I20260620 01:50:38.598719 25240 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20260620 01:50:38.604354 25254 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:38.606673 25240 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:50:38.607019 25240 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "d7ebec69238c4564b5e9a92d34a663cf"
format_stamp: "Formatted at 2026-06-20 01:50:38 on dist-test-slave-wg14"
I20260620 01:50:38.607574 25240 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:38.628471 25240 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:38.629930 25240 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:38.630354 25240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:38.655378 25240 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:35539
I20260620 01:50:38.655431 25305 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:35539 every 8 connection(s)
I20260620 01:50:38.657159 25240 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:50:38.663275 25306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:50:38.669845 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 25240
I20260620 01:50:38.670097 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:50:38.672675 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:46333
--webserver_interface=127.20.1.189
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217 with env {}
I20260620 01:50:38.677431 25306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:38.711808 25306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:50:38.714497 25256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:46333: connect: Connection refused (error 111)
W20260620 01:50:38.717428 25306 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:46333: Network error: Client connection negotiation failed: client connection to 127.20.1.189:46333: connect: Connection refused (error 111)
I20260620 01:50:38.766891 25306 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } attempt: 1
W20260620 01:50:38.769743 25306 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:46333: Network error: Client connection negotiation failed: client connection to 127.20.1.189:46333: connect: Connection refused (error 111)
I20260620 01:50:38.838131 25306 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } attempt: 2
W20260620 01:50:38.840919 25306 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:46333: Network error: Client connection negotiation failed: client connection to 127.20.1.189:46333: connect: Connection refused (error 111)
I20260620 01:50:38.932338 25306 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } attempt: 3
W20260620 01:50:38.934832 25306 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:46333: Network error: Client connection negotiation failed: client connection to 127.20.1.189:46333: connect: Connection refused (error 111)
I20260620 01:50:39.078293 25306 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } attempt: 4
W20260620 01:50:39.080924 25306 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:46333: Network error: Client connection negotiation failed: client connection to 127.20.1.189:46333: connect: Connection refused (error 111)
W20260620 01:50:39.110376 25308 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:39.110822 25308 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:39.110915 25308 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:39.122355 25308 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:39.122494 25308 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:39.122551 25308 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:39.122593 25308 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:39.134248 25308 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:46333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:39.136368 25308 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:39.139376 25308 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:39.151490 25317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:39.151489 25316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:39.152535 25319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:39.154300 25308 server_base.cc:1061] running on GCE node
I20260620 01:50:39.155539 25308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:39.157696 25308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:39.161330 25308 hybrid_clock.cc:648] HybridClock initialized: now 1781920239161082 us; error 152 us; skew 500 ppm
I20260620 01:50:39.161978 25308 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:39.165485 25308 webserver.cc:533] Webserver started at http://127.20.1.189:39087/ using document root <none> and password file <none>
I20260620 01:50:39.166503 25308 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:39.166647 25308 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:39.167070 25308 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:39.170290 25308 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "282c033e2e4b4b0ea77fbf67b4750788"
format_stamp: "Formatted at 2026-06-20 01:50:39 on dist-test-slave-wg14"
I20260620 01:50:39.171517 25308 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "282c033e2e4b4b0ea77fbf67b4750788"
format_stamp: "Formatted at 2026-06-20 01:50:39 on dist-test-slave-wg14"
I20260620 01:50:39.179836 25308 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260620 01:50:39.185689 25325 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:39.188305 25308 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:50:39.188547 25308 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "282c033e2e4b4b0ea77fbf67b4750788"
format_stamp: "Formatted at 2026-06-20 01:50:39 on dist-test-slave-wg14"
I20260620 01:50:39.188769 25308 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:39.207729 25308 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:39.209255 25308 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:39.209656 25308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:39.238438 25308 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:46333
I20260620 01:50:39.238473 25376 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:46333 every 8 connection(s)
I20260620 01:50:39.240381 25308 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:50:39.240916 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 25308
I20260620 01:50:39.241156 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:50:39.242849 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:35217
--webserver_interface=127.20.1.188
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217 with env {}
I20260620 01:50:39.246708 25377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:50:39.255627 25377 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:39.275566 25377 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:39.287883 25377 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:50:39.289649 25327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:35217: connect: Connection refused (error 111)
W20260620 01:50:39.292363 25377 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:35217: Network error: Client connection negotiation failed: client connection to 127.20.1.188:35217: connect: Connection refused (error 111)
I20260620 01:50:39.341791 25377 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } attempt: 1
W20260620 01:50:39.344622 25377 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:35217: Network error: Client connection negotiation failed: client connection to 127.20.1.188:35217: connect: Connection refused (error 111)
I20260620 01:50:39.380317 25306 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } attempt: 5
I20260620 01:50:39.389259 25306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:50:39.392676 25306 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:35217: Network error: Client connection negotiation failed: client connection to 127.20.1.188:35217: connect: Connection refused (error 111)
I20260620 01:50:39.413039 25377 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } attempt: 2
W20260620 01:50:39.415632 25377 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:35217: Network error: Client connection negotiation failed: client connection to 127.20.1.188:35217: connect: Connection refused (error 111)
I20260620 01:50:39.443975 25306 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } attempt: 1
W20260620 01:50:39.446851 25306 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:35217: Network error: Client connection negotiation failed: client connection to 127.20.1.188:35217: connect: Connection refused (error 111)
I20260620 01:50:39.506981 25377 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } attempt: 3
W20260620 01:50:39.510360 25377 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:35217: Network error: Client connection negotiation failed: client connection to 127.20.1.188:35217: connect: Connection refused (error 111)
I20260620 01:50:39.515201 25306 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } attempt: 2
W20260620 01:50:39.518210 25306 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:35217: Network error: Client connection negotiation failed: client connection to 127.20.1.188:35217: connect: Connection refused (error 111)
I20260620 01:50:39.624501 25306 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } attempt: 3
W20260620 01:50:39.628044 25306 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:35217: Network error: Client connection negotiation failed: client connection to 127.20.1.188:35217: connect: Connection refused (error 111)
I20260620 01:50:39.653751 25377 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } attempt: 4
W20260620 01:50:39.656555 25377 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:35217: Network error: Client connection negotiation failed: client connection to 127.20.1.188:35217: connect: Connection refused (error 111)
W20260620 01:50:39.694409 25379 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:39.694852 25379 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:39.694990 25379 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:39.704833 25379 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:39.704975 25379 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:39.705057 25379 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:39.705125 25379 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:39.719302 25379 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:35217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:39.721274 25379 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:39.724545 25379 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:39.737841 25388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:39.738368 25389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:39.739751 25391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:39.739872 25379 server_base.cc:1061] running on GCE node
I20260620 01:50:39.741096 25379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:39.743294 25379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:39.744903 25379 hybrid_clock.cc:648] HybridClock initialized: now 1781920239744680 us; error 127 us; skew 500 ppm
I20260620 01:50:39.745672 25379 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:39.749789 25379 webserver.cc:533] Webserver started at http://127.20.1.188:35483/ using document root <none> and password file <none>
I20260620 01:50:39.750921 25379 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:39.751065 25379 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:39.751524 25379 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:39.754791 25379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "bfe660f20c9b4c598feb923a4536dfe4"
format_stamp: "Formatted at 2026-06-20 01:50:39 on dist-test-slave-wg14"
I20260620 01:50:39.755910 25379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "bfe660f20c9b4c598feb923a4536dfe4"
format_stamp: "Formatted at 2026-06-20 01:50:39 on dist-test-slave-wg14"
I20260620 01:50:39.763418 25379 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 01:50:39.768342 25397 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:39.770371 25379 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 01:50:39.770624 25379 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "bfe660f20c9b4c598feb923a4536dfe4"
format_stamp: "Formatted at 2026-06-20 01:50:39 on dist-test-slave-wg14"
I20260620 01:50:39.770893 25379 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:39.796623 25379 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:39.797689 25379 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:39.798002 25379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:39.805409 25306 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } attempt: 4
W20260620 01:50:39.808359 25306 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:35217: Network error: Client connection negotiation failed: client connection to 127.20.1.188:35217: connect: Connection refused (error 111)
I20260620 01:50:39.825717 25379 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:35217
I20260620 01:50:39.825731 25448 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:35217 every 8 connection(s)
I20260620 01:50:39.827850 25379 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:50:39.830821 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 25379
I20260620 01:50:39.831161 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:50:39.837699 25449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:50:39.847885 25449 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:39.862875 25449 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:39.871062 25449 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:50:39.884208 25449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4: Bootstrap starting.
I20260620 01:50:39.888727 25449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4: Neither blocks nor log segments found. Creating new log.
I20260620 01:50:39.890182 25449 log.cc:826] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:39.894047 25449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4: No bootstrap required, opened a new log
I20260620 01:50:39.900068 25449 raft_consensus.cc:359] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:39.900506 25449 raft_consensus.cc:385] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:50:39.900645 25449 raft_consensus.cc:740] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe660f20c9b4c598feb923a4536dfe4, State: Initialized, Role: FOLLOWER
I20260620 01:50:39.901798 25449 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:39.903152 25456 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:39.903539 25456 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:39.904567 25449 sys_catalog.cc:565] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:50:39.921139 25467 catalog_manager.cc:1593] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:50:39.921357 25467 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:50:39.917393 25379 master_runner.cc:187] Error getting master registration for 127.20.1.190:35539: OK, instance_id { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" instance_seqno: 1781920238648754 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:50:39.924953 25379 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:50:39.955933 25377 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } attempt: 5
I20260620 01:50:39.972012 25377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788: Bootstrap starting.
I20260620 01:50:39.978578 25377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788: Neither blocks nor log segments found. Creating new log.
I20260620 01:50:39.981102 25377 log.cc:826] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:39.984900 25377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788: No bootstrap required, opened a new log
I20260620 01:50:39.991118 25377 raft_consensus.cc:359] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:39.991555 25377 raft_consensus.cc:385] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:50:39.991678 25377 raft_consensus.cc:740] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282c033e2e4b4b0ea77fbf67b4750788, State: Initialized, Role: FOLLOWER
I20260620 01:50:39.993551 25377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:39.996068 25470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:39.996511 25470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:39.997964 25377 sys_catalog.cc:565] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:50:40.015447 25481 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:50:40.015638 25481 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:50:40.011312 25308 master_runner.cc:187] Error getting master registration for 127.20.1.190:35539: OK, instance_id { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" instance_seqno: 1781920238648754 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:50:40.029218 25308 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:50:40.051262 25403 tablet.cc:2406] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:50:40.085742 25306 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } attempt: 5
I20260620 01:50:40.097863 25470 raft_consensus.cc:493] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:50:40.098232 25470 raft_consensus.cc:515] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:40.100553 25470 leader_election.cc:290] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7ebec69238c4564b5e9a92d34a663cf (127.20.1.190:35539), bfe660f20c9b4c598feb923a4536dfe4 (127.20.1.188:35217)
I20260620 01:50:40.109784 25424 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "282c033e2e4b4b0ea77fbf67b4750788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe660f20c9b4c598feb923a4536dfe4" is_pre_election: true
I20260620 01:50:40.111680 25424 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 282c033e2e4b4b0ea77fbf67b4750788 in term 0.
I20260620 01:50:40.113034 25327 leader_election.cc:304] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282c033e2e4b4b0ea77fbf67b4750788, bfe660f20c9b4c598feb923a4536dfe4; no voters:
I20260620 01:50:40.113804 25306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: Bootstrap starting.
I20260620 01:50:40.113976 25470 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:50:40.114161 25470 raft_consensus.cc:493] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:50:40.114307 25470 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:50:40.120494 25470 raft_consensus.cc:515] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:40.125558 25306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: Neither blocks nor log segments found. Creating new log.
I20260620 01:50:40.127708 25306 log.cc:826] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:40.129441 25424 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "282c033e2e4b4b0ea77fbf67b4750788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe660f20c9b4c598feb923a4536dfe4"
I20260620 01:50:40.130133 25424 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:50:40.138187 25470 leader_election.cc:290] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [CANDIDATE]: Term 1 election: Requested vote from peers d7ebec69238c4564b5e9a92d34a663cf (127.20.1.190:35539), bfe660f20c9b4c598feb923a4536dfe4 (127.20.1.188:35217)
I20260620 01:50:40.139189 25306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: No bootstrap required, opened a new log
I20260620 01:50:40.140547 25424 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282c033e2e4b4b0ea77fbf67b4750788 in term 1.
I20260620 01:50:40.141702 25327 leader_election.cc:304] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282c033e2e4b4b0ea77fbf67b4750788, bfe660f20c9b4c598feb923a4536dfe4; no voters:
I20260620 01:50:40.142446 25470 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:50:40.149801 25306 raft_consensus.cc:359] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:40.150535 25306 raft_consensus.cc:385] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:50:40.150755 25306 raft_consensus.cc:740] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ebec69238c4564b5e9a92d34a663cf, State: Initialized, Role: FOLLOWER
I20260620 01:50:40.152134 25306 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:40.152455 25470 raft_consensus.cc:697] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 1 LEADER]: Becoming Leader. State: Replica: 282c033e2e4b4b0ea77fbf67b4750788, State: Running, Role: LEADER
I20260620 01:50:40.153942 25470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:40.159871 25485 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:40.165804 25485 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:40.165627 25281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "282c033e2e4b4b0ea77fbf67b4750788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ebec69238c4564b5e9a92d34a663cf" is_pre_election: true
I20260620 01:50:40.166085 25280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "282c033e2e4b4b0ea77fbf67b4750788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ebec69238c4564b5e9a92d34a663cf"
I20260620 01:50:40.165578 25306 sys_catalog.cc:565] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:40.165046 25486 raft_consensus.cc:493] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:50:40.168473 25486 raft_consensus.cc:515] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:40.169490 25281 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 282c033e2e4b4b0ea77fbf67b4750788 in term 0.
I20260620 01:50:40.176009 25352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7ebec69238c4564b5e9a92d34a663cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282c033e2e4b4b0ea77fbf67b4750788" is_pre_election: true
I20260620 01:50:40.180847 25424 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7ebec69238c4564b5e9a92d34a663cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe660f20c9b4c598feb923a4536dfe4" is_pre_election: true
I20260620 01:50:40.180712 25484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 282c033e2e4b4b0ea77fbf67b4750788. Latest consensus state: current_term: 1 leader_uuid: "282c033e2e4b4b0ea77fbf67b4750788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:40.181180 25484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:40.181288 25424 raft_consensus.cc:2393] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7ebec69238c4564b5e9a92d34a663cf in current term 1: Already voted for candidate 282c033e2e4b4b0ea77fbf67b4750788 in this term.
I20260620 01:50:40.182473 25256 leader_election.cc:304] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7ebec69238c4564b5e9a92d34a663cf; no voters: 282c033e2e4b4b0ea77fbf67b4750788, bfe660f20c9b4c598feb923a4536dfe4
I20260620 01:50:40.183190 25485 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:50:40.184958 25486 leader_election.cc:290] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 282c033e2e4b4b0ea77fbf67b4750788 (127.20.1.189:46333), bfe660f20c9b4c598feb923a4536dfe4 (127.20.1.188:35217)
I20260620 01:50:40.186685 25485 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 01:50:40.195788 25489 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:50:40.198462 25489 catalog_manager.cc:1519] Initializing Kudu cluster ID...
W20260620 01:50:40.214273 25331 tablet.cc:2406] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:50:40.234102 25424 raft_consensus.cc:1275] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 1 FOLLOWER]: Refusing update from remote peer 282c033e2e4b4b0ea77fbf67b4750788: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:50:40.234314 25281 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 1 FOLLOWER]: Refusing update from remote peer 282c033e2e4b4b0ea77fbf67b4750788: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:50:40.235404 25484 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:50:40.235831 25470 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:50:40.264724 25456 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 282c033e2e4b4b0ea77fbf67b4750788. Latest consensus state: current_term: 1 leader_uuid: "282c033e2e4b4b0ea77fbf67b4750788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:40.265143 25456 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:40.283841 25485 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 282c033e2e4b4b0ea77fbf67b4750788. Latest consensus state: current_term: 1 leader_uuid: "282c033e2e4b4b0ea77fbf67b4750788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:40.284168 25485 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:40.304108 25485 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "282c033e2e4b4b0ea77fbf67b4750788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:40.304585 25485 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:40.308038 25500 mvcc.cc:204] Tried to move back new op lower bound from 7298745303975268352 to 7298745303690985472. Current Snapshot: MvccSnapshot[applied={T|T < 7298745303975268352 or (T in {7298745303975268352})}]
I20260620 01:50:40.316526 25456 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "282c033e2e4b4b0ea77fbf67b4750788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:40.316856 25456 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:40.328647 25470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "282c033e2e4b4b0ea77fbf67b4750788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:40.329188 25470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:40.329630 25470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "282c033e2e4b4b0ea77fbf67b4750788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:40.329872 25470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:40.331949 25489 catalog_manager.cc:1382] Generated new cluster ID: e639c3fdbd1841d28fd391b0c9c8f98e
I20260620 01:50:40.332119 25489 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:50:40.358505 25505 catalog_manager.cc:1294] Loaded cluster ID: e639c3fdbd1841d28fd391b0c9c8f98e
I20260620 01:50:40.358644 25505 catalog_manager.cc:1587] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: loading cluster ID for follower catalog manager: success
W20260620 01:50:40.360556 25505 catalog_manager.cc:1611] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260620 01:50:40.360766 25505 catalog_manager.cc:883] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:50:40.370740 25489 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:50:40.372761 25489 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:50:40.408613 25489 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788: Generated new TSK 0
I20260620 01:50:40.410558 25489 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:50:40.419591 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:50:40.924508 25467 catalog_manager.cc:1294] Loaded cluster ID: e639c3fdbd1841d28fd391b0c9c8f98e
I20260620 01:50:40.924677 25467 catalog_manager.cc:1587] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4: loading cluster ID for follower catalog manager: success
I20260620 01:50:40.929316 25467 catalog_manager.cc:1609] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4: acquiring CA information for follower catalog manager: success
I20260620 01:50:40.932698 25467 catalog_manager.cc:1637] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 01:50:41.010828 25510 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:41.011250 25510 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:41.011394 25510 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:41.025450 25510 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:41.025715 25510 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:41.039728 25510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:41.042750 25510 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:41.046044 25510 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:41.060631 25515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:41.060762 25516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:41.062188 25518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:41.063031 25510 server_base.cc:1061] running on GCE node
I20260620 01:50:41.063884 25510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:41.065402 25510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:41.068923 25510 hybrid_clock.cc:648] HybridClock initialized: now 1781920241068673 us; error 175 us; skew 500 ppm
I20260620 01:50:41.069422 25510 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:41.072625 25510 webserver.cc:533] Webserver started at http://127.20.1.129:33205/ using document root <none> and password file <none>
I20260620 01:50:41.073900 25510 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:41.074071 25510 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:41.074780 25510 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:41.078858 25510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "37c73e8242474533961206ee82053e55"
format_stamp: "Formatted at 2026-06-20 01:50:41 on dist-test-slave-wg14"
I20260620 01:50:41.079977 25510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "37c73e8242474533961206ee82053e55"
format_stamp: "Formatted at 2026-06-20 01:50:41 on dist-test-slave-wg14"
I20260620 01:50:41.087194 25510 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 01:50:41.092577 25524 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:41.094882 25510 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:50:41.095223 25510 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "37c73e8242474533961206ee82053e55"
format_stamp: "Formatted at 2026-06-20 01:50:41 on dist-test-slave-wg14"
I20260620 01:50:41.095539 25510 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:41.127593 25510 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:41.128859 25510 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:41.129189 25510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:41.130486 25510 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:41.133208 25510 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:41.133369 25510 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:41.133503 25510 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:41.133733 25510 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:41.179054 25510 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35187
I20260620 01:50:41.179220 25636 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35187 every 8 connection(s)
I20260620 01:50:41.180860 25510 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:41.190127 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 25510
I20260620 01:50:41.190332 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:50:41.195703 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:50:41.205370 25639 heartbeater.cc:344] Connected to a master server at 127.20.1.189:46333
I20260620 01:50:41.206008 25639 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:41.208233 25638 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35539
I20260620 01:50:41.208279 25639 heartbeater.cc:507] Master 127.20.1.189:46333 requested a full tablet report, sending...
I20260620 01:50:41.208766 25638 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:41.209789 25638 heartbeater.cc:507] Master 127.20.1.190:35539 requested a full tablet report, sending...
I20260620 01:50:41.212654 25271 ts_manager.cc:194] Registered new tserver with Master: 37c73e8242474533961206ee82053e55 (127.20.1.129:35187)
I20260620 01:50:41.212656 25341 ts_manager.cc:194] Registered new tserver with Master: 37c73e8242474533961206ee82053e55 (127.20.1.129:35187)
I20260620 01:50:41.216146 25637 heartbeater.cc:344] Connected to a master server at 127.20.1.188:35217
I20260620 01:50:41.216425 25637 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:41.217060 25341 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:42013
I20260620 01:50:41.217528 25637 heartbeater.cc:507] Master 127.20.1.188:35217 requested a full tablet report, sending...
I20260620 01:50:41.220398 25414 ts_manager.cc:194] Registered new tserver with Master: 37c73e8242474533961206ee82053e55 (127.20.1.129:35187)
I20260620 01:50:41.366567 25505 catalog_manager.cc:1609] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: acquiring CA information for follower catalog manager: success
I20260620 01:50:41.369983 25505 catalog_manager.cc:1637] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 01:50:41.645283 25647 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:41.645623 25647 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:41.645720 25647 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:41.655756 25647 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:41.656015 25647 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:50:41.668955 25647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:41.670779 25647 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:41.673223 25647 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:41.689747 25652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:41.689934 25653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:41.690402 25655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:41.692348 25647 server_base.cc:1061] running on GCE node
I20260620 01:50:41.693583 25647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:41.695684 25647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:41.699348 25647 hybrid_clock.cc:648] HybridClock initialized: now 1781920241699069 us; error 204 us; skew 500 ppm
I20260620 01:50:41.699955 25647 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:41.704420 25647 webserver.cc:533] Webserver started at http://127.20.1.130:39595/ using document root <none> and password file <none>
I20260620 01:50:41.705614 25647 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:41.705767 25647 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:41.706190 25647 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:41.709391 25647 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/data/instance:
uuid: "d89c94f002184a7ab10c072928eafff1"
format_stamp: "Formatted at 2026-06-20 01:50:41 on dist-test-slave-wg14"
I20260620 01:50:41.710606 25647 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance:
uuid: "d89c94f002184a7ab10c072928eafff1"
format_stamp: "Formatted at 2026-06-20 01:50:41 on dist-test-slave-wg14"
I20260620 01:50:41.720089 25647 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.011s sys 0.000s
I20260620 01:50:41.727351 25661 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:41.730012 25647 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:50:41.730289 25647 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "d89c94f002184a7ab10c072928eafff1"
format_stamp: "Formatted at 2026-06-20 01:50:41 on dist-test-slave-wg14"
I20260620 01:50:41.730546 25647 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:41.748765 25647 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:41.749883 25647 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:41.750159 25647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:41.751550 25647 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:41.753696 25647 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:41.753836 25647 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:41.753945 25647 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:41.754007 25647 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:41.801517 25647 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:42851
I20260620 01:50:41.801545 25773 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:42851 every 8 connection(s)
I20260620 01:50:41.803386 25647 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:50:41.808023 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 25647
I20260620 01:50:41.808406 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
I20260620 01:50:41.817865 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:50:41.853937 25774 heartbeater.cc:344] Connected to a master server at 127.20.1.188:35217
I20260620 01:50:41.854072 25775 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35539
I20260620 01:50:41.854115 25776 heartbeater.cc:344] Connected to a master server at 127.20.1.189:46333
I20260620 01:50:41.854493 25775 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:41.854488 25776 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:41.854493 25774 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:41.856211 25774 heartbeater.cc:507] Master 127.20.1.188:35217 requested a full tablet report, sending...
I20260620 01:50:41.856274 25775 heartbeater.cc:507] Master 127.20.1.190:35539 requested a full tablet report, sending...
I20260620 01:50:41.856343 25776 heartbeater.cc:507] Master 127.20.1.189:46333 requested a full tablet report, sending...
I20260620 01:50:41.858853 25341 ts_manager.cc:194] Registered new tserver with Master: d89c94f002184a7ab10c072928eafff1 (127.20.1.130:42851)
I20260620 01:50:41.858932 25414 ts_manager.cc:194] Registered new tserver with Master: d89c94f002184a7ab10c072928eafff1 (127.20.1.130:42851)
I20260620 01:50:41.860154 25341 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.130:50823
I20260620 01:50:41.860174 25271 ts_manager.cc:194] Registered new tserver with Master: d89c94f002184a7ab10c072928eafff1 (127.20.1.130:42851)
I20260620 01:50:42.222416 25639 heartbeater.cc:499] Master 127.20.1.189:46333 was elected leader, sending a full tablet report...
W20260620 01:50:42.256981 25784 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:42.257371 25784 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:42.257490 25784 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:42.268352 25784 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:42.268575 25784 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:50:42.280390 25784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:42.282441 25784 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:42.284658 25784 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:42.303619 25790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:42.303541 25789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:42.305768 25784 server_base.cc:1061] running on GCE node
W20260620 01:50:42.306671 25792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:42.307837 25784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:42.309484 25784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:42.313045 25784 hybrid_clock.cc:648] HybridClock initialized: now 1781920242312801 us; error 134 us; skew 500 ppm
I20260620 01:50:42.313670 25784 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:42.317472 25784 webserver.cc:533] Webserver started at http://127.20.1.131:37135/ using document root <none> and password file <none>
I20260620 01:50:42.318495 25784 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:42.318631 25784 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:42.319016 25784 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:50:42.322886 25784 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/data/instance:
uuid: "03999bf2dc4944fa8d3e0cd327bf3280"
format_stamp: "Formatted at 2026-06-20 01:50:42 on dist-test-slave-wg14"
I20260620 01:50:42.324067 25784 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance:
uuid: "03999bf2dc4944fa8d3e0cd327bf3280"
format_stamp: "Formatted at 2026-06-20 01:50:42 on dist-test-slave-wg14"
I20260620 01:50:42.332073 25784 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 01:50:42.337922 25798 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:42.340540 25784 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:50:42.340802 25784 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "03999bf2dc4944fa8d3e0cd327bf3280"
format_stamp: "Formatted at 2026-06-20 01:50:42 on dist-test-slave-wg14"
I20260620 01:50:42.341118 25784 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:42.359331 25784 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:42.360405 25784 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:42.360700 25784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:42.361932 25784 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:42.364763 25784 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:42.364910 25784 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:42.365017 25784 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:42.365177 25784 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:42.408106 25784 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:46531
I20260620 01:50:42.408155 25910 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:46531 every 8 connection(s)
I20260620 01:50:42.409727 25784 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:50:42.415216 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 25784
I20260620 01:50:42.415454 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
I20260620 01:50:42.437434 25912 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35539
I20260620 01:50:42.438038 25912 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:42.439282 25911 heartbeater.cc:344] Connected to a master server at 127.20.1.188:35217
I20260620 01:50:42.439532 25911 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:42.439599 25912 heartbeater.cc:507] Master 127.20.1.190:35539 requested a full tablet report, sending...
I20260620 01:50:42.440236 25911 heartbeater.cc:507] Master 127.20.1.188:35217 requested a full tablet report, sending...
I20260620 01:50:42.442123 25271 ts_manager.cc:194] Registered new tserver with Master: 03999bf2dc4944fa8d3e0cd327bf3280 (127.20.1.131:46531)
I20260620 01:50:42.442130 25414 ts_manager.cc:194] Registered new tserver with Master: 03999bf2dc4944fa8d3e0cd327bf3280 (127.20.1.131:46531)
I20260620 01:50:42.444840 25913 heartbeater.cc:344] Connected to a master server at 127.20.1.189:46333
I20260620 01:50:42.445047 25913 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:42.445592 25913 heartbeater.cc:507] Master 127.20.1.189:46333 requested a full tablet report, sending...
I20260620 01:50:42.447170 25341 ts_manager.cc:194] Registered new tserver with Master: 03999bf2dc4944fa8d3e0cd327bf3280 (127.20.1.131:46531)
I20260620 01:50:42.448442 25341 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.131:60949
I20260620 01:50:42.450266 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:50:42.463663 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:50:42.463899 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 25240
I20260620 01:50:42.465368 25301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:42.581223 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 25240
I20260620 01:50:42.604167 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:35539
--webserver_interface=127.20.1.190
--webserver_port=46771
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217 with env {}
W20260620 01:50:42.784593 25329 consensus_peers.cc:597] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 -> Peer d7ebec69238c4564b5e9a92d34a663cf (127.20.1.190:35539): Couldn't send request to peer d7ebec69238c4564b5e9a92d34a663cf. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:35539: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 01:50:42.863722 25776 heartbeater.cc:499] Master 127.20.1.189:46333 was elected leader, sending a full tablet report...
W20260620 01:50:42.863809 25775 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:35539 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:35539: connect: Connection refused (error 111)
W20260620 01:50:43.024233 25922 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:43.024647 25922 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:43.024777 25922 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:43.035629 25922 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:43.035777 25922 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:43.035844 25922 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:43.035923 25922 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:43.048213 25922 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:35539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=46771
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:43.050210 25922 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:43.053160 25922 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:43.065819 25931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:43.065744 25930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:43.065784 25933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:43.067521 25922 server_base.cc:1061] running on GCE node
I20260620 01:50:43.068606 25922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:43.071033 25922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:43.072702 25922 hybrid_clock.cc:648] HybridClock initialized: now 1781920243072477 us; error 96 us; skew 500 ppm
I20260620 01:50:43.073493 25922 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:43.077313 25922 webserver.cc:533] Webserver started at http://127.20.1.190:46771/ using document root <none> and password file <none>
I20260620 01:50:43.078593 25922 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:43.078748 25922 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:43.086256 25922 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 01:50:43.090502 25939 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:43.092778 25922 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:50:43.093151 25922 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "d7ebec69238c4564b5e9a92d34a663cf"
format_stamp: "Formatted at 2026-06-20 01:50:38 on dist-test-slave-wg14"
I20260620 01:50:43.094131 25922 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:43.118556 25922 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:43.119656 25922 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:43.119979 25922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:43.146027 25990 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:35539 every 8 connection(s)
I20260620 01:50:43.146027 25922 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:35539
I20260620 01:50:43.148499 25922 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:50:43.149284 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 25922
I20260620 01:50:43.149709 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 25510
I20260620 01:50:43.160215 25991 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:50:43.163842 25632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:43.165171 25991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: Bootstrap starting.
I20260620 01:50:43.176891 25470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "282c033e2e4b4b0ea77fbf67b4750788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:43.177486 25470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:43.193344 25991 log.cc:826] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:43.211360 25991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:50:43.212302 25991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: Bootstrap complete.
I20260620 01:50:43.224006 25991 raft_consensus.cc:359] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:43.225346 25991 raft_consensus.cc:740] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ebec69238c4564b5e9a92d34a663cf, State: Initialized, Role: FOLLOWER
I20260620 01:50:43.226509 25991 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:43.227674 26001 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:43.228081 26001 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:43.229362 25991 sys_catalog.cc:565] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:43.230852 26001 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 282c033e2e4b4b0ea77fbf67b4750788. Latest consensus state: current_term: 1 leader_uuid: "282c033e2e4b4b0ea77fbf67b4750788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:43.231518 26001 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:43.256090 26012 catalog_manager.cc:1294] Loaded cluster ID: e639c3fdbd1841d28fd391b0c9c8f98e
I20260620 01:50:43.256356 26012 catalog_manager.cc:1587] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: loading cluster ID for follower catalog manager: success
I20260620 01:50:43.263295 26012 catalog_manager.cc:1609] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: acquiring CA information for follower catalog manager: success
I20260620 01:50:43.271088 26012 catalog_manager.cc:1637] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:50:43.308408 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 25510
W20260620 01:50:43.328143 25328 connection.cc:573] server connection from 127.20.1.129:42013 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 01:50:43.328753 25400 connection.cc:573] server connection from 127.20.1.129:33151 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:50:43.328991 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35187
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--webserver_port=33205
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260620 01:50:43.329807 25999 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:50:43.305884 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:59875 (local address 127.20.1.190:35539)
0620 01:50:43.306071 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0620 01:50:43.306078 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 01:50:43.306717 (+ 639us) server_negotiation.cc:408] Connection header received
0620 01:50:43.307075 (+ 358us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:50:43.307082 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:50:43.307216 (+ 134us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:50:43.307282 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0620 01:50:43.308306 (+ 1024us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:50:43.309871 (+ 1565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:50:43.328226 (+ 18355us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:59875: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 01:50:43.452068 25913 heartbeater.cc:499] Master 127.20.1.189:46333 was elected leader, sending a full tablet report...
I20260620 01:50:43.458792 25956 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "03999bf2dc4944fa8d3e0cd327bf3280" instance_seqno: 1781920242398098) as {username='slave'} at 127.20.1.131:59711; Asking this server to re-register.
I20260620 01:50:43.460191 25912 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:43.462932 25956 ts_manager.cc:194] Registered new tserver with Master: 03999bf2dc4944fa8d3e0cd327bf3280 (127.20.1.131:46531)
I20260620 01:50:43.558141 25997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "282c033e2e4b4b0ea77fbf67b4750788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:43.558593 25997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: This master's current role is: LEADER
W20260620 01:50:43.764986 26018 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:43.765345 26018 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:43.765432 26018 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:43.774626 26018 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:43.774849 26018 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:43.786906 26018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35187
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=33205
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:43.789058 26018 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:43.791533 26018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:43.806408 26024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:43.806423 26025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:43.807760 26027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:43.809543 26018 server_base.cc:1061] running on GCE node
I20260620 01:50:43.810443 26018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:43.811925 26018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:43.813282 26018 hybrid_clock.cc:648] HybridClock initialized: now 1781920243813117 us; error 120 us; skew 500 ppm
I20260620 01:50:43.813687 26018 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:43.816648 26018 webserver.cc:533] Webserver started at http://127.20.1.129:33205/ using document root <none> and password file <none>
I20260620 01:50:43.817595 26018 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:43.817749 26018 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:43.825089 26018 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:50:43.829246 26033 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:43.831602 26018 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260620 01:50:43.831841 26018 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "37c73e8242474533961206ee82053e55"
format_stamp: "Formatted at 2026-06-20 01:50:41 on dist-test-slave-wg14"
I20260620 01:50:43.833117 26018 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:43.873241 25775 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35539
I20260620 01:50:43.874298 26018 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:43.874963 25956 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d89c94f002184a7ab10c072928eafff1" instance_seqno: 1781920241789605) as {username='slave'} at 127.20.1.130:58669; Asking this server to re-register.
I20260620 01:50:43.875638 26018 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:43.875718 25775 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:43.875939 26018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:43.877133 26018 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:43.877143 25956 ts_manager.cc:194] Registered new tserver with Master: d89c94f002184a7ab10c072928eafff1 (127.20.1.130:42851)
I20260620 01:50:43.879161 26018 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:43.879283 26018 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 01:50:43.879468 26018 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:43.879546 26018 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:43.921960 26018 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35187
I20260620 01:50:43.922011 26146 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35187 every 8 connection(s)
I20260620 01:50:43.924574 26018 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:43.929390 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 26018
I20260620 01:50:43.929828 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 26018
I20260620 01:50:43.959581 26147 heartbeater.cc:344] Connected to a master server at 127.20.1.188:35217
I20260620 01:50:43.959699 26149 heartbeater.cc:344] Connected to a master server at 127.20.1.189:46333
I20260620 01:50:43.960167 26147 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:43.959653 26148 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35539
I20260620 01:50:43.960459 26148 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:43.960170 26149 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:43.961970 26148 heartbeater.cc:507] Master 127.20.1.190:35539 requested a full tablet report, sending...
I20260620 01:50:43.962086 26147 heartbeater.cc:507] Master 127.20.1.188:35217 requested a full tablet report, sending...
I20260620 01:50:43.962296 26149 heartbeater.cc:507] Master 127.20.1.189:46333 requested a full tablet report, sending...
I20260620 01:50:43.965068 25956 ts_manager.cc:194] Registered new tserver with Master: 37c73e8242474533961206ee82053e55 (127.20.1.129:35187)
I20260620 01:50:43.965214 25341 ts_manager.cc:194] Re-registered known tserver with Master: 37c73e8242474533961206ee82053e55 (127.20.1.129:35187)
I20260620 01:50:43.966455 25341 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:48349
I20260620 01:50:43.968017 25414 ts_manager.cc:194] Re-registered known tserver with Master: 37c73e8242474533961206ee82053e55 (127.20.1.129:35187)
I20260620 01:50:43.970122 26142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:44.071265 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 26018
I20260620 01:50:44.086630 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35187
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--webserver_port=33205
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
W20260620 01:50:44.531738 26159 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:44.532143 26159 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:44.532467 26159 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:50:44.532595 26159 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:44.542856 26159 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:44.543272 26159 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:44.555574 26159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35187
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=33205
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:44.558039 26159 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:44.560523 26159 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:44.575157 26164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:44.575096 26165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:44.575204 26167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:44.576813 26159 server_base.cc:1061] running on GCE node
I20260620 01:50:44.577687 26159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:44.579372 26159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:44.583278 26159 hybrid_clock.cc:648] HybridClock initialized: now 1781920244582978 us; error 106 us; skew 500 ppm
I20260620 01:50:44.583802 26159 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:44.587747 26159 webserver.cc:533] Webserver started at http://127.20.1.129:33205/ using document root <none> and password file <none>
I20260620 01:50:44.588881 26159 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:44.589107 26159 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:44.597277 26159 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 01:50:44.602085 26173 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:44.604146 26159 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:50:44.604424 26159 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "37c73e8242474533961206ee82053e55"
format_stamp: "Formatted at 2026-06-20 01:50:41 on dist-test-slave-wg14"
I20260620 01:50:44.605548 26159 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:44.629745 26159 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:44.630904 26159 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:44.631278 26159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:44.632704 26159 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:44.635382 26159 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:44.635552 26159 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:44.635686 26159 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:44.635771 26159 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:50:44.636368 26159 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:50:44.636482 26159 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:50:44.636596 26159 fault_injection.cc:59] injecting fault for kudu (pid 26159): FLAGS_fault_before_start (process will exit)
I20260620 01:50:44.657226 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35187
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--webserver_port=33205
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260620 01:50:45.129418 26181 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:45.130349 26181 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:45.130616 26181 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:45.141041 26181 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:45.141279 26181 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:45.161146 26181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:35349,127.20.1.149:44425,127.20.1.150:43301,127.20.1.151:36125,127.20.1.152:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35187
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=33205
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:45.163501 26181 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:45.166623 26181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:45.181321 26189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:45.181380 26191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:45.181329 26188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:45.183594 26181 server_base.cc:1061] running on GCE node
I20260620 01:50:45.184382 26181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:45.187458 26181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:45.191021 26181 hybrid_clock.cc:648] HybridClock initialized: now 1781920245190683 us; error 261 us; skew 500 ppm
I20260620 01:50:45.191610 26181 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:45.195246 26181 webserver.cc:533] Webserver started at http://127.20.1.129:33205/ using document root <none> and password file <none>
I20260620 01:50:45.196312 26181 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:45.196487 26181 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:45.210014 26181 fs_manager.cc:714] Time spent opening directory manager: real 0.011s user 0.006s sys 0.000s
I20260620 01:50:45.216077 26197 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:45.220664 26181 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.002s sys 0.002s
I20260620 01:50:45.220921 26181 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "37c73e8242474533961206ee82053e55"
format_stamp: "Formatted at 2026-06-20 01:50:41 on dist-test-slave-wg14"
I20260620 01:50:45.222890 26181 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:45.291076 26181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:45.292462 26181 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:45.292812 26181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:45.294384 26181 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:45.296914 26181 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:45.297039 26181 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:45.297152 26181 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:45.297231 26181 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:45.350435 26181 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35187
I20260620 01:50:45.350436 26309 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35187 every 8 connection(s)
I20260620 01:50:45.353106 26181 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:45.359818 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 26181
I20260620 01:50:45.360213 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 26181
I20260620 01:50:45.383868 26312 heartbeater.cc:344] Connected to a master server at 127.20.1.189:46333
I20260620 01:50:45.384735 26312 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:45.386919 26312 heartbeater.cc:507] Master 127.20.1.189:46333 requested a full tablet report, sending...
I20260620 01:50:45.389365 26311 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35539
I20260620 01:50:45.390174 26311 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:45.392402 25341 ts_manager.cc:194] Re-registered known tserver with Master: 37c73e8242474533961206ee82053e55 (127.20.1.129:35187)
I20260620 01:50:45.393111 26311 heartbeater.cc:507] Master 127.20.1.190:35539 requested a full tablet report, sending...
I20260620 01:50:45.394106 25341 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:57175
I20260620 01:50:45.396183 25956 ts_manager.cc:194] Re-registered known tserver with Master: 37c73e8242474533961206ee82053e55 (127.20.1.129:35187)
I20260620 01:50:45.397933 26310 heartbeater.cc:344] Connected to a master server at 127.20.1.188:35217
I20260620 01:50:45.398128 26310 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:45.398816 26310 heartbeater.cc:507] Master 127.20.1.188:35217 requested a full tablet report, sending...
I20260620 01:50:45.398895 26305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:45.508280 25414 ts_manager.cc:194] Re-registered known tserver with Master: 37c73e8242474533961206ee82053e55 (127.20.1.129:35187)
I20260620 01:50:45.508517 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 26181
W20260620 01:50:45.527107 25399 connection.cc:573] server connection from 127.20.1.129:39209 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:50:45.528342 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 25647
I20260620 01:50:45.538895 25769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:45.643515 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 25647
I20260620 01:50:45.662628 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 25784
I20260620 01:50:45.674661 25906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:45.779693 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 25784
I20260620 01:50:45.803248 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 25922
I20260620 01:50:45.816534 25986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:45.963065 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 25922
I20260620 01:50:45.988050 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 25308
I20260620 01:50:45.989703 25372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:46.128443 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 25308
I20260620 01:50:46.148643 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 25379
I20260620 01:50:46.150394 25444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:46.279613 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 25379
2026-06-20T01:50:46Z chronyd exiting
2026-06-20T01:50:46Z chronyd exiting
2026-06-20T01:50:46Z chronyd exiting
2026-06-20T01:50:46Z chronyd exiting
2026-06-20T01:50:46Z chronyd exiting
2026-06-20T01:50:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:50:46Z Disabled control of system clock
2026-06-20T01:50:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:50:46Z Disabled control of system clock
2026-06-20T01:50:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:50:46Z Disabled control of system clock
2026-06-20T01:50:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:50:46Z Disabled control of system clock
2026-06-20T01:50:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:50:46Z Disabled control of system clock
I20260620 01:50:46.490933 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:35539
--webserver_interface=127.20.1.190
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217 with env {}
W20260620 01:50:46.921183 26340 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:46.921602 26340 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:46.921726 26340 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:46.933028 26340 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:46.933215 26340 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:46.933279 26340 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:46.933341 26340 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:46.945051 26340 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:35539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:46.946816 26340 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:46.949066 26340 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:46.959177 26346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:46.959215 26345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:46.961620 26348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:46.961843 26340 server_base.cc:1061] running on GCE node
I20260620 01:50:46.962958 26340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:46.965605 26340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:46.969314 26340 hybrid_clock.cc:648] HybridClock initialized: now 1781920246969074 us; error 156 us; skew 500 ppm
I20260620 01:50:46.969874 26340 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:46.973193 26340 webserver.cc:533] Webserver started at http://127.20.1.190:33637/ using document root <none> and password file <none>
I20260620 01:50:46.974412 26340 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:46.974588 26340 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:46.982784 26340 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 01:50:46.987598 26354 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:46.989508 26340 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:50:46.989741 26340 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "d7ebec69238c4564b5e9a92d34a663cf"
format_stamp: "Formatted at 2026-06-20 01:50:38 on dist-test-slave-wg14"
I20260620 01:50:46.990794 26340 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:47.024907 26340 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:47.025985 26340 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:47.026327 26340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:47.055611 26340 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:35539
I20260620 01:50:47.055717 26405 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:35539 every 8 connection(s)
I20260620 01:50:47.058035 26340 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:50:47.058799 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 26340
I20260620 01:50:47.059019 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:50:47.060689 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:46333
--webserver_interface=127.20.1.189
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217 with env {}
I20260620 01:50:47.073129 26406 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:50:47.077342 26406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: Bootstrap starting.
I20260620 01:50:47.100783 26406 log.cc:826] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:47.113567 26406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:50:47.114665 26406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: Bootstrap complete.
I20260620 01:50:47.123248 26406 raft_consensus.cc:359] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:47.124136 26406 raft_consensus.cc:740] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ebec69238c4564b5e9a92d34a663cf, State: Initialized, Role: FOLLOWER
I20260620 01:50:47.125273 26406 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:47.126428 26410 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:47.126919 26410 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:47.128345 26406 sys_catalog.cc:565] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:47.148432 26421 catalog_manager.cc:1294] Loaded cluster ID: e639c3fdbd1841d28fd391b0c9c8f98e
I20260620 01:50:47.150650 26421 catalog_manager.cc:1587] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: loading cluster ID for follower catalog manager: success
I20260620 01:50:47.157534 26421 catalog_manager.cc:1609] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: acquiring CA information for follower catalog manager: success
I20260620 01:50:47.162352 26421 catalog_manager.cc:1637] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:50:47.184733 26340 master_runner.cc:187] Error getting master registration for 127.20.1.189:46333: Network error: Client connection negotiation failed: client connection to 127.20.1.189:46333: connect: Connection refused (error 111),
I20260620 01:50:47.192771 26340 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:50:47.513645 26408 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:47.514061 26408 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:47.514279 26408 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:47.525056 26408 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:47.525182 26408 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:47.525238 26408 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:47.525278 26408 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:47.537736 26408 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:46333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:47.539693 26408 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:47.541877 26408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:47.554822 26430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:47.555960 26429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:47.557400 26432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:47.558094 26408 server_base.cc:1061] running on GCE node
I20260620 01:50:47.559417 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:47.562463 26408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:47.566287 26408 hybrid_clock.cc:648] HybridClock initialized: now 1781920247565912 us; error 154 us; skew 500 ppm
I20260620 01:50:47.567013 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:47.570709 26408 webserver.cc:533] Webserver started at http://127.20.1.189:36057/ using document root <none> and password file <none>
I20260620 01:50:47.571977 26408 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:47.572186 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:47.582129 26408 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.003s
I20260620 01:50:47.587352 26438 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:47.590227 26408 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260620 01:50:47.590540 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "282c033e2e4b4b0ea77fbf67b4750788"
format_stamp: "Formatted at 2026-06-20 01:50:39 on dist-test-slave-wg14"
I20260620 01:50:47.591758 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:47.646284 26408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:47.647413 26408 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:47.647713 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:47.672362 26408 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:46333
I20260620 01:50:47.672391 26489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:46333 every 8 connection(s)
I20260620 01:50:47.676299 26408 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:50:47.686822 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 26408
I20260620 01:50:47.687242 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:50:47.688390 26490 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:50:47.688529 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:35217
--webserver_interface=127.20.1.188
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217 with env {}
I20260620 01:50:47.692238 26490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788: Bootstrap starting.
I20260620 01:50:47.719084 26490 log.cc:826] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:47.734771 26490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:50:47.735647 26490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788: Bootstrap complete.
I20260620 01:50:47.746750 26490 raft_consensus.cc:359] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:47.748006 26490 raft_consensus.cc:740] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282c033e2e4b4b0ea77fbf67b4750788, State: Initialized, Role: FOLLOWER
I20260620 01:50:47.749231 26490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:47.750427 26494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:47.751963 26490 sys_catalog.cc:565] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:47.752050 26494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:47.769395 26505 catalog_manager.cc:1294] Loaded cluster ID: e639c3fdbd1841d28fd391b0c9c8f98e
I20260620 01:50:47.769654 26505 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788: loading cluster ID for follower catalog manager: success
I20260620 01:50:47.775055 26505 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788: acquiring CA information for follower catalog manager: success
I20260620 01:50:47.778779 26505 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:50:47.798252 26408 master_runner.cc:187] Error getting master registration for 127.20.1.188:35217: Network error: Client connection negotiation failed: client connection to 127.20.1.188:35217: connect: Connection refused (error 111),
I20260620 01:50:47.802300 26408 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:50:48.190945 26492 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:48.191455 26492 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:48.191610 26492 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:48.202649 26492 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:48.202808 26492 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:48.202898 26492 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:48.202970 26492 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:48.217147 26492 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:35217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:48.219849 26492 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:48.222443 26492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:48.232676 26515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:48.234782 26514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:48.237391 26517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:48.237232 26492 server_base.cc:1061] running on GCE node
I20260620 01:50:48.238741 26492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:48.241513 26492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:48.245080 26492 hybrid_clock.cc:648] HybridClock initialized: now 1781920248244808 us; error 195 us; skew 500 ppm
I20260620 01:50:48.245709 26492 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:48.248477 26492 webserver.cc:533] Webserver started at http://127.20.1.188:37897/ using document root <none> and password file <none>
I20260620 01:50:48.249292 26492 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:48.249823 26492 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:48.258641 26492 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:50:48.270617 26523 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:48.275555 26492 fs_manager.cc:730] Time spent opening block manager: real 0.014s user 0.002s sys 0.010s
I20260620 01:50:48.275871 26492 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "bfe660f20c9b4c598feb923a4536dfe4"
format_stamp: "Formatted at 2026-06-20 01:50:39 on dist-test-slave-wg14"
I20260620 01:50:48.276830 26492 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:48.292169 26492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:48.293223 26492 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:48.293565 26492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:48.319468 26574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:35217 every 8 connection(s)
I20260620 01:50:48.319470 26492 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:35217
I20260620 01:50:48.321736 26492 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:50:48.322769 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 26492
I20260620 01:50:48.323017 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:50:48.336017 26575 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:50:48.343294 26575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4: Bootstrap starting.
I20260620 01:50:48.367467 26575 log.cc:826] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:48.381096 26575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:50:48.381739 26575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4: Bootstrap complete.
I20260620 01:50:48.389490 26575 raft_consensus.cc:359] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:48.390298 26575 raft_consensus.cc:740] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe660f20c9b4c598feb923a4536dfe4, State: Initialized, Role: FOLLOWER
I20260620 01:50:48.391057 26575 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:48.392427 26582 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:48.392951 26582 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:48.394963 26575 sys_catalog.cc:565] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:48.410912 26593 catalog_manager.cc:1294] Loaded cluster ID: e639c3fdbd1841d28fd391b0c9c8f98e
I20260620 01:50:48.411078 26593 catalog_manager.cc:1587] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4: loading cluster ID for follower catalog manager: success
I20260620 01:50:48.415673 26593 catalog_manager.cc:1609] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4: acquiring CA information for follower catalog manager: success
I20260620 01:50:48.420603 26593 catalog_manager.cc:1637] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:50:48.445258 26492 master_runner.cc:264] No leader master found from master bfe660f20c9b4c598feb923a4536dfe4
I20260620 01:50:48.445504 26492 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:50:48.857775 26596 raft_consensus.cc:493] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:50:48.858368 26596 raft_consensus.cc:515] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:48.861280 26596 leader_election.cc:290] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 282c033e2e4b4b0ea77fbf67b4750788 (127.20.1.189:46333), bfe660f20c9b4c598feb923a4536dfe4 (127.20.1.188:35217)
I20260620 01:50:48.871798 26465 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7ebec69238c4564b5e9a92d34a663cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "282c033e2e4b4b0ea77fbf67b4750788" is_pre_election: true
I20260620 01:50:48.872465 26550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7ebec69238c4564b5e9a92d34a663cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bfe660f20c9b4c598feb923a4536dfe4" is_pre_election: true
I20260620 01:50:48.872699 26465 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ebec69238c4564b5e9a92d34a663cf in term 1.
I20260620 01:50:48.873181 26550 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ebec69238c4564b5e9a92d34a663cf in term 1.
I20260620 01:50:48.873874 26356 leader_election.cc:304] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282c033e2e4b4b0ea77fbf67b4750788, d7ebec69238c4564b5e9a92d34a663cf; no voters:
I20260620 01:50:48.874498 26596 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:50:48.874948 26596 raft_consensus.cc:493] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:50:48.875159 26596 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:50:48.879488 26596 raft_consensus.cc:515] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:48.880319 26596 leader_election.cc:290] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [CANDIDATE]: Term 2 election: Requested vote from peers 282c033e2e4b4b0ea77fbf67b4750788 (127.20.1.189:46333), bfe660f20c9b4c598feb923a4536dfe4 (127.20.1.188:35217)
I20260620 01:50:48.881354 26465 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7ebec69238c4564b5e9a92d34a663cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "282c033e2e4b4b0ea77fbf67b4750788"
I20260620 01:50:48.881606 26465 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:50:48.881511 26550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7ebec69238c4564b5e9a92d34a663cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bfe660f20c9b4c598feb923a4536dfe4"
I20260620 01:50:48.881970 26550 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:50:48.886790 26550 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ebec69238c4564b5e9a92d34a663cf in term 2.
I20260620 01:50:48.886821 26465 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ebec69238c4564b5e9a92d34a663cf in term 2.
I20260620 01:50:48.887702 26356 leader_election.cc:304] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfe660f20c9b4c598feb923a4536dfe4, d7ebec69238c4564b5e9a92d34a663cf; no voters:
I20260620 01:50:48.888221 26596 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 2 FOLLOWER]: Leader election won for term 2
I20260620 01:50:48.888974 26596 raft_consensus.cc:697] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 2 LEADER]: Becoming Leader. State: Replica: d7ebec69238c4564b5e9a92d34a663cf, State: Running, Role: LEADER
I20260620 01:50:48.889577 26596 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:48.892784 26599 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7ebec69238c4564b5e9a92d34a663cf. Latest consensus state: current_term: 2 leader_uuid: "d7ebec69238c4564b5e9a92d34a663cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:48.893165 26599 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:48.893927 26601 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:50:48.896349 26601 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:50:48.897336 26601 catalog_manager.cc:1294] Loaded cluster ID: e639c3fdbd1841d28fd391b0c9c8f98e
I20260620 01:50:48.897442 26601 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:50:48.898900 26601 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:50:48.900027 26601 catalog_manager.cc:6080] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: Loaded TSK: 0
I20260620 01:50:48.901254 26601 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:50:48.926506 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:50:49.343434 26465 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 2 FOLLOWER]: Refusing update from remote peer d7ebec69238c4564b5e9a92d34a663cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:50:49.344673 26599 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:50:49.350010 26603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7ebec69238c4564b5e9a92d34a663cf. Latest consensus state: current_term: 2 leader_uuid: "d7ebec69238c4564b5e9a92d34a663cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:49.350603 26603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:49.352613 26599 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d7ebec69238c4564b5e9a92d34a663cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:49.352936 26599 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:49.354998 26550 raft_consensus.cc:1275] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 2 FOLLOWER]: Refusing update from remote peer d7ebec69238c4564b5e9a92d34a663cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:50:49.356324 26596 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:50:49.356566 26603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d7ebec69238c4564b5e9a92d34a663cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:49.356848 26603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:49.362823 26608 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7ebec69238c4564b5e9a92d34a663cf. Latest consensus state: current_term: 2 leader_uuid: "d7ebec69238c4564b5e9a92d34a663cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:49.363365 26608 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:49.362613 26611 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d7ebec69238c4564b5e9a92d34a663cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:49.363619 26611 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:50:49.391443 26602 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:49.391804 26602 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:49.391904 26602 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:49.403975 26602 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:49.404584 26602 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:49.419270 26602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:49.421321 26602 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:49.423682 26602 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:49.439204 26616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:49.439164 26617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:49.439217 26619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:49.440907 26602 server_base.cc:1061] running on GCE node
I20260620 01:50:49.441874 26602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:49.443838 26602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:49.447381 26602 hybrid_clock.cc:648] HybridClock initialized: now 1781920249447156 us; error 151 us; skew 500 ppm
I20260620 01:50:49.447855 26602 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:49.451632 26602 webserver.cc:533] Webserver started at http://127.20.1.129:34459/ using document root <none> and password file <none>
I20260620 01:50:49.452782 26602 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:49.452908 26602 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:49.461720 26602 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:50:49.466064 26625 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:49.468720 26602 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:50:49.468995 26602 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "37c73e8242474533961206ee82053e55"
format_stamp: "Formatted at 2026-06-20 01:50:41 on dist-test-slave-wg14"
I20260620 01:50:49.470331 26602 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:49.486621 26602 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:49.488080 26602 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:49.488452 26602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:49.490120 26602 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:49.492487 26602 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:49.492604 26602 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:49.492723 26602 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:49.492801 26602 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:49.539413 26602 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:36153
I20260620 01:50:49.539712 26737 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:36153 every 8 connection(s)
I20260620 01:50:49.541571 26602 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:49.549171 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 26602
I20260620 01:50:49.549489 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:50:49.556821 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:50:49.570379 26739 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35539
I20260620 01:50:49.570639 26738 heartbeater.cc:344] Connected to a master server at 127.20.1.188:35217
I20260620 01:50:49.570977 26738 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:49.570982 26739 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:49.573011 26738 heartbeater.cc:507] Master 127.20.1.188:35217 requested a full tablet report, sending...
I20260620 01:50:49.573230 26739 heartbeater.cc:507] Master 127.20.1.190:35539 requested a full tablet report, sending...
I20260620 01:50:49.577152 26371 ts_manager.cc:194] Registered new tserver with Master: 37c73e8242474533961206ee82053e55 (127.20.1.129:36153)
I20260620 01:50:49.577152 26540 ts_manager.cc:194] Registered new tserver with Master: 37c73e8242474533961206ee82053e55 (127.20.1.129:36153)
I20260620 01:50:49.579648 26371 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:56085
I20260620 01:50:49.580293 26740 heartbeater.cc:344] Connected to a master server at 127.20.1.189:46333
I20260620 01:50:49.580533 26740 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:49.581212 26740 heartbeater.cc:507] Master 127.20.1.189:46333 requested a full tablet report, sending...
I20260620 01:50:49.584205 26455 ts_manager.cc:194] Registered new tserver with Master: 37c73e8242474533961206ee82053e55 (127.20.1.129:36153)
I20260620 01:50:49.960148 26596 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d7ebec69238c4564b5e9a92d34a663cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:49.960523 26596 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: This master's current role is: LEADER
W20260620 01:50:50.009958 26748 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:50.010519 26748 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:50.010865 26748 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:50.024314 26748 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:50.024569 26748 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:50:50.039065 26748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:50.042927 26748 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:50.045876 26748 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:50.060596 26756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:50.062901 26757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:50.066752 26759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:50.069342 26748 server_base.cc:1061] running on GCE node
I20260620 01:50:50.070180 26748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:50.071732 26748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:50.075348 26748 hybrid_clock.cc:648] HybridClock initialized: now 1781920250075093 us; error 167 us; skew 500 ppm
I20260620 01:50:50.075834 26748 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:50.079525 26748 webserver.cc:533] Webserver started at http://127.20.1.130:40421/ using document root <none> and password file <none>
I20260620 01:50:50.080750 26748 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:50.080950 26748 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:50.089423 26748 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 01:50:50.094158 26765 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:50.096760 26748 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260620 01:50:50.097056 26748 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "d89c94f002184a7ab10c072928eafff1"
format_stamp: "Formatted at 2026-06-20 01:50:41 on dist-test-slave-wg14"
I20260620 01:50:50.098148 26748 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:50.123725 26748 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:50.125161 26748 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:50.125774 26748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:50.127465 26748 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:50.129824 26748 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:50.129972 26748 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:50.130129 26748 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:50.130187 26748 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:50.171370 26748 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:45735
I20260620 01:50:50.171422 26877 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:45735 every 8 connection(s)
I20260620 01:50:50.174295 26748 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:50:50.174816 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 26748
I20260620 01:50:50.175465 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
I20260620 01:50:50.183455 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:50:50.204510 26879 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35539
I20260620 01:50:50.205097 26879 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:50.206754 26879 heartbeater.cc:507] Master 127.20.1.190:35539 requested a full tablet report, sending...
I20260620 01:50:50.207217 26878 heartbeater.cc:344] Connected to a master server at 127.20.1.188:35217
I20260620 01:50:50.207393 26878 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:50.208029 26878 heartbeater.cc:507] Master 127.20.1.188:35217 requested a full tablet report, sending...
I20260620 01:50:50.210709 26540 ts_manager.cc:194] Registered new tserver with Master: d89c94f002184a7ab10c072928eafff1 (127.20.1.130:45735)
I20260620 01:50:50.211002 26371 ts_manager.cc:194] Registered new tserver with Master: d89c94f002184a7ab10c072928eafff1 (127.20.1.130:45735)
I20260620 01:50:50.211683 26880 heartbeater.cc:344] Connected to a master server at 127.20.1.189:46333
I20260620 01:50:50.211860 26880 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:50.212884 26880 heartbeater.cc:507] Master 127.20.1.189:46333 requested a full tablet report, sending...
I20260620 01:50:50.212956 26371 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.130:39239
I20260620 01:50:50.215090 26455 ts_manager.cc:194] Registered new tserver with Master: d89c94f002184a7ab10c072928eafff1 (127.20.1.130:45735)
I20260620 01:50:50.583308 26739 heartbeater.cc:499] Master 127.20.1.190:35539 was elected leader, sending a full tablet report...
W20260620 01:50:50.647330 26886 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:50.647769 26886 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:50.647928 26886 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:50.660141 26886 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:50.660365 26886 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:50:50.672572 26886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:50.674942 26886 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:50.677703 26886 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:50.692494 26893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:50.692516 26896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:50.692564 26894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:50.694427 26886 server_base.cc:1061] running on GCE node
I20260620 01:50:50.695333 26886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:50.696902 26886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:50.700340 26886 hybrid_clock.cc:648] HybridClock initialized: now 1781920250700076 us; error 189 us; skew 500 ppm
I20260620 01:50:50.701009 26886 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:50.705433 26886 webserver.cc:533] Webserver started at http://127.20.1.131:39233/ using document root <none> and password file <none>
I20260620 01:50:50.707540 26886 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:50.707904 26886 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:50.716255 26886 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:50:50.720955 26902 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:50.723744 26886 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:50:50.724543 26886 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "03999bf2dc4944fa8d3e0cd327bf3280"
format_stamp: "Formatted at 2026-06-20 01:50:42 on dist-test-slave-wg14"
I20260620 01:50:50.726487 26886 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:50.745496 26886 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:50.747367 26886 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:50.747814 26886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:50.749871 26886 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:50.753223 26886 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:50.753357 26886 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:50.753523 26886 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:50.753584 26886 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:50.799744 26886 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:35743
I20260620 01:50:50.799804 27014 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:35743 every 8 connection(s)
I20260620 01:50:50.801563 26886 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:50:50.811734 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 26886
I20260620 01:50:50.811970 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
I20260620 01:50:50.824990 27015 heartbeater.cc:344] Connected to a master server at 127.20.1.188:35217
I20260620 01:50:50.825647 27015 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:50.826902 27015 heartbeater.cc:507] Master 127.20.1.188:35217 requested a full tablet report, sending...
I20260620 01:50:50.828140 27016 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35539
I20260620 01:50:50.828382 27016 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:50.829164 27016 heartbeater.cc:507] Master 127.20.1.190:35539 requested a full tablet report, sending...
I20260620 01:50:50.829561 26540 ts_manager.cc:194] Registered new tserver with Master: 03999bf2dc4944fa8d3e0cd327bf3280 (127.20.1.131:35743)
I20260620 01:50:50.830484 26371 ts_manager.cc:194] Registered new tserver with Master: 03999bf2dc4944fa8d3e0cd327bf3280 (127.20.1.131:35743)
I20260620 01:50:50.831662 26371 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.131:37509
I20260620 01:50:50.832943 27017 heartbeater.cc:344] Connected to a master server at 127.20.1.189:46333
I20260620 01:50:50.833138 27017 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:50.833999 27017 heartbeater.cc:507] Master 127.20.1.189:46333 requested a full tablet report, sending...
I20260620 01:50:50.835366 26455 ts_manager.cc:194] Registered new tserver with Master: 03999bf2dc4944fa8d3e0cd327bf3280 (127.20.1.131:35743)
I20260620 01:50:50.845999 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:50:50.860601 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:50:50.860868 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 26340
I20260620 01:50:50.862295 26401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:50.993214 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 26340
I20260620 01:50:51.017735 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:35539
--webserver_interface=127.20.1.190
--webserver_port=33637
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217 with env {}
I20260620 01:50:51.218743 26879 heartbeater.cc:499] Master 127.20.1.190:35539 was elected leader, sending a full tablet report...
W20260620 01:50:51.221225 26879 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:35539 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:35539: connect: Connection refused (error 111)
W20260620 01:50:51.467309 27027 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:51.467798 27027 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:51.467981 27027 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:51.479326 27027 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:50:51.479454 27027 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:51.479529 27027 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:50:51.479597 27027 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:50:51.493543 27027 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:35539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=33637
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:51.495651 27027 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:51.498178 27027 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:51.508896 27033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:51.508994 27034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:51.510890 27036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:51.512991 27027 server_base.cc:1061] running on GCE node
I20260620 01:50:51.514676 27027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:51.517725 27027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:51.521292 27027 hybrid_clock.cc:648] HybridClock initialized: now 1781920251521048 us; error 153 us; skew 500 ppm
I20260620 01:50:51.522311 27027 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:51.525395 27027 webserver.cc:533] Webserver started at http://127.20.1.190:33637/ using document root <none> and password file <none>
I20260620 01:50:51.526861 27027 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:51.527098 27027 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:51.537062 27027 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.003s sys 0.005s
I20260620 01:50:51.542294 27042 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:51.545136 27027 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260620 01:50:51.545445 27027 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "d7ebec69238c4564b5e9a92d34a663cf"
format_stamp: "Formatted at 2026-06-20 01:50:38 on dist-test-slave-wg14"
I20260620 01:50:51.546501 27027 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:51.572007 27027 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:51.573565 27027 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:51.573952 27027 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260620 01:50:51.591385 26739 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:35539 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:35539: connect: Connection refused (error 111)
I20260620 01:50:51.608038 27027 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:35539
I20260620 01:50:51.608067 27094 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:35539 every 8 connection(s)
I20260620 01:50:51.610575 27027 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:50:51.618100 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 27027
I20260620 01:50:51.618548 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 26602
I20260620 01:50:51.621065 27095 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:50:51.628453 27095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: Bootstrap starting.
I20260620 01:50:51.644114 26733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:51.655117 27095 log.cc:826] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: Log is configured to *not* fsync() on all Append() calls
I20260620 01:50:51.669344 27095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:50:51.670012 27095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: Bootstrap complete.
I20260620 01:50:51.679949 27095 raft_consensus.cc:359] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:51.681118 27095 raft_consensus.cc:740] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ebec69238c4564b5e9a92d34a663cf, State: Initialized, Role: FOLLOWER
I20260620 01:50:51.682394 27095 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:51.686218 27101 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:51.686741 27101 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:51.687690 27095 sys_catalog.cc:565] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:50:51.709901 27112 catalog_manager.cc:1294] Loaded cluster ID: e639c3fdbd1841d28fd391b0c9c8f98e
I20260620 01:50:51.710060 27112 catalog_manager.cc:1587] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: loading cluster ID for follower catalog manager: success
I20260620 01:50:51.727479 27112 catalog_manager.cc:1609] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: acquiring CA information for follower catalog manager: success
I20260620 01:50:51.730878 27112 catalog_manager.cc:1637] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:50:51.792502 27027 master_runner.cc:264] No leader master found from master d7ebec69238c4564b5e9a92d34a663cf
I20260620 01:50:51.792670 27027 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:50:51.816004 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 26602
I20260620 01:50:51.832353 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:36153
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--webserver_port=34459
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:50:51.834934 27016 heartbeater.cc:499] Master 127.20.1.190:35539 was elected leader, sending a full tablet report...
I20260620 01:50:51.851065 27059 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "03999bf2dc4944fa8d3e0cd327bf3280" instance_seqno: 1781920250790268) as {username='slave'} at 127.20.1.131:58875; Asking this server to re-register.
I20260620 01:50:51.852188 27016 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:51.854508 27059 ts_manager.cc:194] Registered new tserver with Master: 03999bf2dc4944fa8d3e0cd327bf3280 (127.20.1.131:35743)
I20260620 01:50:52.232805 26879 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35539
I20260620 01:50:52.233145 26879 heartbeater.cc:499] Master 127.20.1.190:35539 was elected leader, sending a full tablet report...
I20260620 01:50:52.234398 27059 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d89c94f002184a7ab10c072928eafff1" instance_seqno: 1781920250161456) as {username='slave'} at 127.20.1.130:33985; Asking this server to re-register.
I20260620 01:50:52.235287 26879 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:52.237010 27059 ts_manager.cc:194] Registered new tserver with Master: d89c94f002184a7ab10c072928eafff1 (127.20.1.130:45735)
W20260620 01:50:52.319167 27119 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:52.319651 27119 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:52.319813 27119 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:52.329784 27119 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:52.330014 27119 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:52.344332 27119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:36153
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=34459
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:52.346585 27119 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:52.349093 27119 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:52.365576 27125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:52.365561 27126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:52.366243 27128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:52.368057 27119 server_base.cc:1061] running on GCE node
I20260620 01:50:52.368964 27119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:52.370496 27119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:52.372045 27119 hybrid_clock.cc:648] HybridClock initialized: now 1781920252371880 us; error 113 us; skew 500 ppm
I20260620 01:50:52.372445 27119 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:52.376945 27119 webserver.cc:533] Webserver started at http://127.20.1.129:34459/ using document root <none> and password file <none>
I20260620 01:50:52.378268 27119 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:52.378425 27119 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:52.387535 27119 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 01:50:52.391909 27134 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:52.394410 27119 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260620 01:50:52.394688 27119 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "37c73e8242474533961206ee82053e55"
format_stamp: "Formatted at 2026-06-20 01:50:41 on dist-test-slave-wg14"
I20260620 01:50:52.395993 27119 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:52.412985 27119 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:52.414112 27119 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:52.414460 27119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:52.415580 27119 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:52.417632 27119 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:52.417775 27119 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:52.417915 27119 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:52.417991 27119 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:52.461802 27119 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:36153
I20260620 01:50:52.461997 27246 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:36153 every 8 connection(s)
I20260620 01:50:52.464524 27119 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:52.474222 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 27119
I20260620 01:50:52.474704 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 27119
I20260620 01:50:52.496027 27259 raft_consensus.cc:493] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7ebec69238c4564b5e9a92d34a663cf)
I20260620 01:50:52.496721 27259 raft_consensus.cc:515] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:52.508145 27258 raft_consensus.cc:493] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7ebec69238c4564b5e9a92d34a663cf)
I20260620 01:50:52.508562 27258 raft_consensus.cc:515] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:52.510556 27259 leader_election.cc:290] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7ebec69238c4564b5e9a92d34a663cf (127.20.1.190:35539), bfe660f20c9b4c598feb923a4536dfe4 (127.20.1.188:35217)
I20260620 01:50:52.511430 27242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:52.511529 27258 leader_election.cc:290] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7ebec69238c4564b5e9a92d34a663cf (127.20.1.190:35539), 282c033e2e4b4b0ea77fbf67b4750788 (127.20.1.189:46333)
I20260620 01:50:52.520776 26465 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfe660f20c9b4c598feb923a4536dfe4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "282c033e2e4b4b0ea77fbf67b4750788" is_pre_election: true
I20260620 01:50:52.521425 26465 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe660f20c9b4c598feb923a4536dfe4 in term 2.
I20260620 01:50:52.522348 26525 leader_election.cc:304] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282c033e2e4b4b0ea77fbf67b4750788, bfe660f20c9b4c598feb923a4536dfe4; no voters:
I20260620 01:50:52.523828 27258 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 01:50:52.523991 27258 raft_consensus.cc:493] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7ebec69238c4564b5e9a92d34a663cf)
I20260620 01:50:52.524091 27258 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:50:52.527807 27258 raft_consensus.cc:515] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:52.531281 26550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "282c033e2e4b4b0ea77fbf67b4750788" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bfe660f20c9b4c598feb923a4536dfe4" is_pre_election: true
I20260620 01:50:52.531598 27258 leader_election.cc:290] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [CANDIDATE]: Term 3 election: Requested vote from peers d7ebec69238c4564b5e9a92d34a663cf (127.20.1.190:35539), 282c033e2e4b4b0ea77fbf67b4750788 (127.20.1.189:46333)
I20260620 01:50:52.531719 26550 raft_consensus.cc:2393] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 282c033e2e4b4b0ea77fbf67b4750788 in current term 3: Already voted for candidate bfe660f20c9b4c598feb923a4536dfe4 in this term.
I20260620 01:50:52.532352 26465 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfe660f20c9b4c598feb923a4536dfe4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "282c033e2e4b4b0ea77fbf67b4750788"
I20260620 01:50:52.532868 26465 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:50:52.536528 26465 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe660f20c9b4c598feb923a4536dfe4 in term 3.
I20260620 01:50:52.537719 26525 leader_election.cc:304] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282c033e2e4b4b0ea77fbf67b4750788, bfe660f20c9b4c598feb923a4536dfe4; no voters:
I20260620 01:50:52.539208 27258 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 01:50:52.542518 27070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "282c033e2e4b4b0ea77fbf67b4750788" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7ebec69238c4564b5e9a92d34a663cf" is_pre_election: true
I20260620 01:50:52.543341 27070 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 282c033e2e4b4b0ea77fbf67b4750788 in term 2.
I20260620 01:50:52.544094 26442 leader_election.cc:304] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 282c033e2e4b4b0ea77fbf67b4750788, d7ebec69238c4564b5e9a92d34a663cf; no voters: bfe660f20c9b4c598feb923a4536dfe4
I20260620 01:50:52.544445 27070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfe660f20c9b4c598feb923a4536dfe4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7ebec69238c4564b5e9a92d34a663cf" is_pre_election: true
I20260620 01:50:52.544649 27070 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe660f20c9b4c598feb923a4536dfe4 in term 2.
I20260620 01:50:52.544946 27259 raft_consensus.cc:2764] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260620 01:50:52.545001 27069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfe660f20c9b4c598feb923a4536dfe4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7ebec69238c4564b5e9a92d34a663cf"
I20260620 01:50:52.545228 27069 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:50:52.549072 27258 raft_consensus.cc:697] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [term 3 LEADER]: Becoming Leader. State: Replica: bfe660f20c9b4c598feb923a4536dfe4, State: Running, Role: LEADER
I20260620 01:50:52.549835 27258 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } }
I20260620 01:50:52.551887 27069 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe660f20c9b4c598feb923a4536dfe4 in term 3.
I20260620 01:50:52.554296 27267 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfe660f20c9b4c598feb923a4536dfe4. Latest consensus state: current_term: 3 leader_uuid: "bfe660f20c9b4c598feb923a4536dfe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:52.554699 27267 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:52.555613 27269 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:50:52.559027 27269 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:50:52.560488 27269 catalog_manager.cc:1294] Loaded cluster ID: e639c3fdbd1841d28fd391b0c9c8f98e
I20260620 01:50:52.560603 27269 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:50:52.561779 27269 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:50:52.562645 27269 catalog_manager.cc:6080] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4: Loaded TSK: 0
I20260620 01:50:52.563972 27269 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:50:52.636009 27247 heartbeater.cc:344] Connected to a master server at 127.20.1.188:35217
I20260620 01:50:52.636643 27247 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:52.639900 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 27119
I20260620 01:50:52.657384 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:36153
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--webserver_port=34459
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
W20260620 01:50:52.657827 27257 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:50:52.483175 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:34381 (local address 127.20.1.189:46333)
0620 01:50:52.488963 (+ 5788us) server_negotiation.cc:207] Beginning negotiation
0620 01:50:52.488971 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:50:52.498189 (+ 9218us) server_negotiation.cc:408] Connection header received
0620 01:50:52.498295 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:50:52.498304 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:50:52.498415 (+ 111us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:50:52.498561 (+ 146us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:50:52.500487 (+ 1926us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:50:52.518008 (+ 17521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:50:52.656489 (+138481us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:34381: BlockingRecv error: recv got EOF from 127.20.1.129:34381 (error 108)
Metrics: {"server-negotiator.queue_time_us":5273,"thread_start_us":5413,"threads_started":1}
W20260620 01:50:52.660176 27114 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:50:52.470889 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:44293 (local address 127.20.1.190:35539)
0620 01:50:52.471105 (+ 216us) server_negotiation.cc:207] Beginning negotiation
0620 01:50:52.471112 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 01:50:52.488175 (+ 17063us) server_negotiation.cc:408] Connection header received
0620 01:50:52.489396 (+ 1221us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:50:52.489404 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:50:52.489512 (+ 108us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:50:52.489593 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:50:52.492137 (+ 2544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:50:52.494231 (+ 2094us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:50:52.640711 (+146480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:50:52.641718 (+ 1007us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0620 01:50:52.659027 (+ 17309us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:44293: BlockingRecv error: recv got EOF from 127.20.1.129:44293 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 01:50:52.986481 26465 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [term 3 FOLLOWER]: Refusing update from remote peer bfe660f20c9b4c598feb923a4536dfe4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:50:52.987907 27267 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:50:52.991677 27259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfe660f20c9b4c598feb923a4536dfe4. Latest consensus state: current_term: 3 leader_uuid: "bfe660f20c9b4c598feb923a4536dfe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:52.992082 27259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:52.995908 27258 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "bfe660f20c9b4c598feb923a4536dfe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:52.996307 27258 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: This master's current role is: LEADER
I20260620 01:50:52.999053 27069 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [term 3 FOLLOWER]: Refusing update from remote peer bfe660f20c9b4c598feb923a4536dfe4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:50:53.000985 27258 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:50:53.005331 27259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bfe660f20c9b4c598feb923a4536dfe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:53.005705 27259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282c033e2e4b4b0ea77fbf67b4750788 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:53.011879 27278 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bfe660f20c9b4c598feb923a4536dfe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:53.012486 27278 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:50:53.012754 27275 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfe660f20c9b4c598feb923a4536dfe4. Latest consensus state: current_term: 3 leader_uuid: "bfe660f20c9b4c598feb923a4536dfe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:53.013150 27275 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ebec69238c4564b5e9a92d34a663cf [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:50:53.110998 27270 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:53.111542 27270 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:53.111683 27270 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:50:53.111763 27270 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:53.121470 27270 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:53.121826 27270 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:53.134680 27270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:36153
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=34459
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:53.137650 27270 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:53.140733 27270 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:53.154870 27286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:53.154870 27284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:53.155496 27283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:53.156008 27270 server_base.cc:1061] running on GCE node
I20260620 01:50:53.156891 27270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:53.158444 27270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:53.162000 27270 hybrid_clock.cc:648] HybridClock initialized: now 1781920253161614 us; error 293 us; skew 500 ppm
I20260620 01:50:53.162510 27270 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:53.166141 27270 webserver.cc:533] Webserver started at http://127.20.1.129:34459/ using document root <none> and password file <none>
I20260620 01:50:53.167228 27270 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:53.167415 27270 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:53.175426 27270 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:50:53.179816 27292 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:53.182502 27270 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:50:53.182844 27270 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "37c73e8242474533961206ee82053e55"
format_stamp: "Formatted at 2026-06-20 01:50:41 on dist-test-slave-wg14"
I20260620 01:50:53.184581 27270 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:53.229518 27270 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:53.230532 27270 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:53.230811 27270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:53.232085 27270 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:53.234836 27270 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:53.235107 27270 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:53.235476 27270 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:53.235626 27270 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:50:53.236338 27270 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:50:53.236497 27270 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:50:53.236567 27270 fault_injection.cc:59] injecting fault for kudu (pid 27270): FLAGS_fault_before_start (process will exit)
I20260620 01:50:53.257695 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:36153
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--webserver_port=34459
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:50:53.434352 27267 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "bfe660f20c9b4c598feb923a4536dfe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ebec69238c4564b5e9a92d34a663cf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35539 } } peers { permanent_uuid: "282c033e2e4b4b0ea77fbf67b4750788" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46333 } } peers { permanent_uuid: "bfe660f20c9b4c598feb923a4536dfe4" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 35217 } } }
I20260620 01:50:53.434715 27267 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfe660f20c9b4c598feb923a4536dfe4 [sys.catalog]: This master's current role is: LEADER
W20260620 01:50:53.713464 27300 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:50:53.713828 27300 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:50:53.713924 27300 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:50:53.724100 27300 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:50:53.724452 27300 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:50:53.737866 27300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39115,127.20.1.149:43745,127.20.1.150:36623,127.20.1.151:45207,127.20.1.152:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:36153
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=34459
--tserver_master_addrs=127.20.1.190:35539,127.20.1.189:46333,127.20.1.188:35217
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:50:53.740352 27300 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:50:53.742692 27300 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:50:53.755394 27306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:53.755409 27305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:50:53.756163 27308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:50:53.757818 27300 server_base.cc:1061] running on GCE node
I20260620 01:50:53.758934 27300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:50:53.761436 27300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:50:53.765146 27300 hybrid_clock.cc:648] HybridClock initialized: now 1781920253764893 us; error 162 us; skew 500 ppm
I20260620 01:50:53.765688 27300 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:50:53.769112 27300 webserver.cc:533] Webserver started at http://127.20.1.129:34459/ using document root <none> and password file <none>
I20260620 01:50:53.770061 27300 fs_manager.cc:362] Metadata directory not provided
I20260620 01:50:53.770200 27300 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:50:53.777060 27300 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 01:50:53.780843 27314 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:53.783296 27300 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260620 01:50:53.783574 27300 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "37c73e8242474533961206ee82053e55"
format_stamp: "Formatted at 2026-06-20 01:50:41 on dist-test-slave-wg14"
I20260620 01:50:53.784688 27300 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:50:53.811728 27300 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:50:53.812783 27300 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:50:53.813113 27300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:50:53.814380 27300 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:50:53.816570 27300 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:50:53.816706 27300 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:53.816880 27300 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:50:53.816972 27300 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:50:53.838891 27015 heartbeater.cc:499] Master 127.20.1.188:35217 was elected leader, sending a full tablet report...
I20260620 01:50:53.861963 27300 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:36153
I20260620 01:50:53.862053 27426 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:36153 every 8 connection(s)
I20260620 01:50:53.863718 27300 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:50:53.867059 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 27300
I20260620 01:50:53.867460 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 27300
I20260620 01:50:53.903699 27422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:54.005077 27428 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35539
I20260620 01:50:54.006069 27428 heartbeater.cc:461] Registering TS with master...
I20260620 01:50:54.005210 27429 heartbeater.cc:344] Connected to a master server at 127.20.1.189:46333
I20260620 01:50:54.007622 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 27300
I20260620 01:50:54.007777 27429 heartbeater.cc:461] Registering TS with master...
W20260620 01:50:54.024478 26524 connection.cc:573] server connection from 127.20.1.129:48355 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:50:54.025208 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 26748
I20260620 01:50:54.037624 26873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:54.146018 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 26748
I20260620 01:50:54.163262 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 26886
I20260620 01:50:54.174479 27010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:54.285494 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 26886
I20260620 01:50:54.303417 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 27027
I20260620 01:50:54.313686 27090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:54.448534 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 27027
I20260620 01:50:54.469755 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 26408
I20260620 01:50:54.471470 26485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:54.607827 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 26408
I20260620 01:50:54.627683 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 26492
I20260620 01:50:54.629383 26570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:50:54.761986 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 26492
2026-06-20T01:50:54Z chronyd exiting
2026-06-20T01:50:54Z chronyd exiting
2026-06-20T01:50:54Z chronyd exiting
2026-06-20T01:50:54Z chronyd exiting
2026-06-20T01:50:54Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/2 (16976 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/3
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/security.properties
2026-06-20 01:51:05: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260620 01:51:18.706287 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44385
--webserver_interface=127.20.1.190
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989 with env {}
W20260620 01:51:19.164667 27732 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:19.165246 27732 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:19.165423 27732 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:19.178591 27732 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:51:19.178755 27732 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:19.178848 27732 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:51:19.178916 27732 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:51:19.192206 27732 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:19.196204 27732 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:19.198755 27732 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:19.215965 27738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:19.216665 27737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:19.217043 27740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:19.217312 27732 server_base.cc:1061] running on GCE node
I20260620 01:51:19.218966 27732 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:19.219290 27732 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:19.219419 27732 hybrid_clock.cc:648] HybridClock initialized: now 1781920279219390 us; error 0 us; skew 500 ppm
I20260620 01:51:19.220052 27732 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:19.232303 27732 webserver.cc:533] Webserver started at http://127.20.1.190:40955/ using document root <none> and password file <none>
I20260620 01:51:19.235102 27732 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:19.235548 27732 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:19.236225 27732 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:51:19.240790 27732 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "e50045e01d1f47b9819454f38124309a"
format_stamp: "Formatted at 2026-06-20 01:51:19 on dist-test-slave-wg14"
I20260620 01:51:19.242520 27732 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "e50045e01d1f47b9819454f38124309a"
format_stamp: "Formatted at 2026-06-20 01:51:19 on dist-test-slave-wg14"
I20260620 01:51:19.256114 27732 fs_manager.cc:696] Time spent creating directory manager: real 0.013s user 0.011s sys 0.000s
I20260620 01:51:19.260917 27745 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:19.264353 27732 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:51:19.264707 27732 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "e50045e01d1f47b9819454f38124309a"
format_stamp: "Formatted at 2026-06-20 01:51:19 on dist-test-slave-wg14"
I20260620 01:51:19.265326 27732 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:19.301715 27732 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:19.302995 27732 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:19.303356 27732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:19.333220 27732 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44385
I20260620 01:51:19.333232 27796 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44385 every 8 connection(s)
I20260620 01:51:19.335636 27732 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:51:19.340035 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 27732
I20260620 01:51:19.340334 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:51:19.341961 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:33769
--webserver_interface=127.20.1.189
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989 with env {}
I20260620 01:51:19.343699 27797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:51:19.352136 27797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:51:19.381333 27797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:51:19.383039 27747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:33769: connect: Connection refused (error 111)
W20260620 01:51:19.385457 27797 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:33769: Network error: Client connection negotiation failed: client connection to 127.20.1.189:33769: connect: Connection refused (error 111)
I20260620 01:51:19.434964 27797 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } attempt: 1
W20260620 01:51:19.438139 27797 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:33769: Network error: Client connection negotiation failed: client connection to 127.20.1.189:33769: connect: Connection refused (error 111)
I20260620 01:51:19.506479 27797 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } attempt: 2
W20260620 01:51:19.510658 27797 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:33769: Network error: Client connection negotiation failed: client connection to 127.20.1.189:33769: connect: Connection refused (error 111)
I20260620 01:51:19.602061 27797 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } attempt: 3
W20260620 01:51:19.604795 27797 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:33769: Network error: Client connection negotiation failed: client connection to 127.20.1.189:33769: connect: Connection refused (error 111)
I20260620 01:51:19.748155 27797 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } attempt: 4
W20260620 01:51:19.751451 27797 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:33769: Network error: Client connection negotiation failed: client connection to 127.20.1.189:33769: connect: Connection refused (error 111)
W20260620 01:51:19.810096 27799 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:19.810570 27799 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:19.810663 27799 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:19.821023 27799 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:51:19.821171 27799 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:19.821241 27799 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:51:19.821316 27799 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:51:19.834712 27799 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:33769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:19.837666 27799 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:19.840548 27799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:19.852865 27807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:19.853027 27808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:19.853425 27810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:19.855558 27799 server_base.cc:1061] running on GCE node
I20260620 01:51:19.856822 27799 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:19.857300 27799 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:19.857441 27799 hybrid_clock.cc:648] HybridClock initialized: now 1781920279857415 us; error 0 us; skew 500 ppm
I20260620 01:51:19.857930 27799 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:19.862370 27799 webserver.cc:533] Webserver started at http://127.20.1.189:41579/ using document root <none> and password file <none>
I20260620 01:51:19.863464 27799 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:19.863742 27799 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:19.864403 27799 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:51:19.867918 27799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "324f486c9a9043f2b98e035d3d5528d2"
format_stamp: "Formatted at 2026-06-20 01:51:19 on dist-test-slave-wg14"
I20260620 01:51:19.869081 27799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "324f486c9a9043f2b98e035d3d5528d2"
format_stamp: "Formatted at 2026-06-20 01:51:19 on dist-test-slave-wg14"
I20260620 01:51:19.877874 27799 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.000s sys 0.008s
I20260620 01:51:19.883833 27815 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:19.886376 27799 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260620 01:51:19.886695 27799 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "324f486c9a9043f2b98e035d3d5528d2"
format_stamp: "Formatted at 2026-06-20 01:51:19 on dist-test-slave-wg14"
I20260620 01:51:19.887012 27799 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:19.907454 27799 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:19.908635 27799 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:19.909293 27799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:19.940253 27866 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:33769 every 8 connection(s)
I20260620 01:51:19.940321 27799 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:33769
I20260620 01:51:19.942483 27799 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:51:19.945979 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 27799
I20260620 01:51:19.946234 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:51:19.947706 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:43989
--webserver_interface=127.20.1.188
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989 with env {}
I20260620 01:51:19.949113 27867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:51:19.960237 27867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:51:19.982218 27867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:51:20.000773 27867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:51:20.002672 27819 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43989: connect: Connection refused (error 111)
W20260620 01:51:20.005728 27867 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43989: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43989: connect: Connection refused (error 111)
I20260620 01:51:20.050907 27797 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } attempt: 5
I20260620 01:51:20.055475 27867 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } attempt: 1
W20260620 01:51:20.058295 27867 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43989: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43989: connect: Connection refused (error 111)
I20260620 01:51:20.062893 27797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:51:20.074568 27797 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43989: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43989: connect: Connection refused (error 111)
I20260620 01:51:20.125943 27797 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } attempt: 1
I20260620 01:51:20.126614 27867 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } attempt: 2
W20260620 01:51:20.128970 27867 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43989: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43989: connect: Connection refused (error 111)
W20260620 01:51:20.129010 27797 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43989: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43989: connect: Connection refused (error 111)
I20260620 01:51:20.197446 27797 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } attempt: 2
W20260620 01:51:20.200220 27797 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43989: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43989: connect: Connection refused (error 111)
I20260620 01:51:20.220330 27867 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } attempt: 3
W20260620 01:51:20.223287 27867 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43989: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43989: connect: Connection refused (error 111)
I20260620 01:51:20.306804 27797 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } attempt: 3
W20260620 01:51:20.310833 27797 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43989: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43989: connect: Connection refused (error 111)
I20260620 01:51:20.366698 27867 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } attempt: 4
W20260620 01:51:20.370049 27867 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43989: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43989: connect: Connection refused (error 111)
W20260620 01:51:20.465626 27869 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:20.466027 27869 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:20.466138 27869 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:20.475823 27869 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:51:20.475960 27869 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:20.476042 27869 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:51:20.476107 27869 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:51:20.488310 27797 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } attempt: 4
I20260620 01:51:20.489766 27869 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:43989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
W20260620 01:51:20.492190 27797 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43989: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43989: connect: Connection refused (error 111)
I20260620 01:51:20.492400 27869 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:20.496232 27869 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:20.510443 27878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:20.510519 27879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:20.514526 27881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:20.516079 27869 server_base.cc:1061] running on GCE node
I20260620 01:51:20.517454 27869 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:20.517731 27869 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:20.517859 27869 hybrid_clock.cc:648] HybridClock initialized: now 1781920280517833 us; error 0 us; skew 500 ppm
I20260620 01:51:20.518355 27869 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:20.522919 27869 webserver.cc:533] Webserver started at http://127.20.1.188:45007/ using document root <none> and password file <none>
I20260620 01:51:20.524061 27869 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:20.524207 27869 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:20.524605 27869 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:51:20.528131 27869 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "7acb459ab1bb42aab827fd0d715bc4a2"
format_stamp: "Formatted at 2026-06-20 01:51:20 on dist-test-slave-wg14"
I20260620 01:51:20.529258 27869 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "7acb459ab1bb42aab827fd0d715bc4a2"
format_stamp: "Formatted at 2026-06-20 01:51:20 on dist-test-slave-wg14"
I20260620 01:51:20.537122 27869 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260620 01:51:20.541954 27886 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:20.544538 27869 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:51:20.544837 27869 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "7acb459ab1bb42aab827fd0d715bc4a2"
format_stamp: "Formatted at 2026-06-20 01:51:20 on dist-test-slave-wg14"
I20260620 01:51:20.545156 27869 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:20.568495 27869 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:20.569938 27869 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:20.570366 27869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:20.606971 27869 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:43989
I20260620 01:51:20.607089 27937 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:43989 every 8 connection(s)
I20260620 01:51:20.609232 27869 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:51:20.612951 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 27869
I20260620 01:51:20.613219 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:51:20.620744 27938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:51:20.635947 27938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:51:20.655719 27938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:51:20.663921 27938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:51:20.669417 27867 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } attempt: 5
I20260620 01:51:20.684309 27938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2: Bootstrap starting.
I20260620 01:51:20.692643 27867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2: Bootstrap starting.
I20260620 01:51:20.693557 27938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2: Neither blocks nor log segments found. Creating new log.
I20260620 01:51:20.695572 27938 log.cc:826] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2: Log is configured to *not* fsync() on all Append() calls
I20260620 01:51:20.698292 27867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2: Neither blocks nor log segments found. Creating new log.
I20260620 01:51:20.699671 27938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2: No bootstrap required, opened a new log
I20260620 01:51:20.699894 27867 log.cc:826] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2: Log is configured to *not* fsync() on all Append() calls
I20260620 01:51:20.702769 27867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2: No bootstrap required, opened a new log
I20260620 01:51:20.706944 27938 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:20.707526 27938 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:51:20.707736 27938 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7acb459ab1bb42aab827fd0d715bc4a2, State: Initialized, Role: FOLLOWER
I20260620 01:51:20.708683 27938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:20.710096 27948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:20.710718 27948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:20.711971 27938 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:51:20.713466 27867 raft_consensus.cc:359] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:20.714579 27867 raft_consensus.cc:385] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:51:20.714869 27867 raft_consensus.cc:740] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324f486c9a9043f2b98e035d3d5528d2, State: Initialized, Role: FOLLOWER
I20260620 01:51:20.715886 27867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:20.717360 27954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:20.718008 27954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:20.718478 27867 sys_catalog.cc:565] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:51:20.731310 27961 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:51:20.733887 27961 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:51:20.728720 27869 master_runner.cc:187] Error getting master registration for 127.20.1.190:44385: OK, instance_id { permanent_uuid: "e50045e01d1f47b9819454f38124309a" instance_seqno: 1781920279324801 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:51:20.746366 27869 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:51:20.756551 27972 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:51:20.757107 27972 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:51:20.746735 27799 master_runner.cc:187] Error getting master registration for 127.20.1.190:44385: OK, instance_id { permanent_uuid: "e50045e01d1f47b9819454f38124309a" instance_seqno: 1781920279324801 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:51:20.758198 27799 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:51:20.769495 27797 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } attempt: 5
I20260620 01:51:20.794540 27797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: Bootstrap starting.
I20260620 01:51:20.800576 27797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: Neither blocks nor log segments found. Creating new log.
I20260620 01:51:20.802492 27797 log.cc:826] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: Log is configured to *not* fsync() on all Append() calls
I20260620 01:51:20.811419 27797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: No bootstrap required, opened a new log
W20260620 01:51:20.823794 27892 tablet.cc:2406] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:51:20.829344 27797 raft_consensus.cc:359] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:20.829928 27797 raft_consensus.cc:385] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:51:20.830248 27797 raft_consensus.cc:740] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50045e01d1f47b9819454f38124309a, State: Initialized, Role: FOLLOWER
I20260620 01:51:20.834828 27797 consensus_queue.cc:260] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:20.836939 27975 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:20.837693 27975 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:20.838884 27797 sys_catalog.cc:565] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:51:20.880888 27948 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:51:20.881337 27948 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:20.883754 27948 leader_election.cc:290] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e50045e01d1f47b9819454f38124309a (127.20.1.190:44385), 324f486c9a9043f2b98e035d3d5528d2 (127.20.1.189:33769)
I20260620 01:51:20.885600 27842 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "324f486c9a9043f2b98e035d3d5528d2" is_pre_election: true
I20260620 01:51:20.885936 27772 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e50045e01d1f47b9819454f38124309a" is_pre_election: true
I20260620 01:51:20.886281 27842 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7acb459ab1bb42aab827fd0d715bc4a2 in term 0.
I20260620 01:51:20.886510 27772 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7acb459ab1bb42aab827fd0d715bc4a2 in term 0.
I20260620 01:51:20.887467 27888 leader_election.cc:304] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 324f486c9a9043f2b98e035d3d5528d2, 7acb459ab1bb42aab827fd0d715bc4a2; no voters:
I20260620 01:51:20.888172 27948 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:51:20.888434 27948 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:51:20.888628 27948 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:51:20.892395 27948 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:20.899423 27732 master_runner.cc:233] Existing masters have differing terms: 0 vs 1
I20260620 01:51:20.899914 27732 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:51:20.900789 27948 leader_election.cc:290] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [CANDIDATE]: Term 1 election: Requested vote from peers e50045e01d1f47b9819454f38124309a (127.20.1.190:44385), 324f486c9a9043f2b98e035d3d5528d2 (127.20.1.189:33769)
I20260620 01:51:20.901678 27842 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "324f486c9a9043f2b98e035d3d5528d2"
I20260620 01:51:20.902001 27772 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e50045e01d1f47b9819454f38124309a"
I20260620 01:51:20.902050 27842 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:51:20.902280 27772 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:51:20.906335 27842 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7acb459ab1bb42aab827fd0d715bc4a2 in term 1.
I20260620 01:51:20.907339 27888 leader_election.cc:304] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 324f486c9a9043f2b98e035d3d5528d2, 7acb459ab1bb42aab827fd0d715bc4a2; no voters:
I20260620 01:51:20.907651 27772 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7acb459ab1bb42aab827fd0d715bc4a2 in term 1.
I20260620 01:51:20.908380 27948 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 1 FOLLOWER]: Leader election won for term 1
W20260620 01:51:20.911815 27986 catalog_manager.cc:1593] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:51:20.912149 27986 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 01:51:20.914140 27821 tablet.cc:2406] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:51:20.931607 27948 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 1 LEADER]: Becoming Leader. State: Replica: 7acb459ab1bb42aab827fd0d715bc4a2, State: Running, Role: LEADER
I20260620 01:51:20.932622 27948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:20.946924 27988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7acb459ab1bb42aab827fd0d715bc4a2. Latest consensus state: current_term: 1 leader_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:20.947376 27988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: This master's current role is: LEADER
I20260620 01:51:20.956250 27990 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:51:20.959921 27990 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:51:20.983613 27842 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 1 FOLLOWER]: Refusing update from remote peer 7acb459ab1bb42aab827fd0d715bc4a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:51:20.986114 27988 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:51:20.992673 27772 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 1 FOLLOWER]: Refusing update from remote peer 7acb459ab1bb42aab827fd0d715bc4a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:51:20.994413 27988 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:51:21.021377 27975 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7acb459ab1bb42aab827fd0d715bc4a2. Latest consensus state: current_term: 1 leader_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:21.021848 27975 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:21.028518 27991 mvcc.cc:204] Tried to move back new op lower bound from 7298745470888230912 to 7298745470710542336. Current Snapshot: MvccSnapshot[applied={T|T < 7298745470888230912}]
I20260620 01:51:21.030663 27954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7acb459ab1bb42aab827fd0d715bc4a2. Latest consensus state: current_term: 1 leader_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:21.032904 27954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:21.035449 27948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:21.037171 27948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: This master's current role is: LEADER
I20260620 01:51:21.048048 27975 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:21.048417 27975 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:21.050757 27990 catalog_manager.cc:1382] Generated new cluster ID: 18a4b0e44a1a4a1497f4c91cfd7df64b
I20260620 01:51:21.051095 27990 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:51:21.064976 27948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:21.065399 27948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: This master's current role is: LEADER
I20260620 01:51:21.068511 27992 mvcc.cc:204] Tried to move back new op lower bound from 7298745470888230912 to 7298745470710542336. Current Snapshot: MvccSnapshot[applied={T|T < 7298745470888230912}]
I20260620 01:51:21.078171 27954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:21.078527 27954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:21.135445 27990 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:51:21.138013 27990 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:51:21.171120 27990 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2: Generated new TSK 0
I20260620 01:51:21.173012 27990 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:51:21.185098 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--time_source=system_unsync with env {}
I20260620 01:51:21.759403 27972 catalog_manager.cc:1294] Loaded cluster ID: 18a4b0e44a1a4a1497f4c91cfd7df64b
I20260620 01:51:21.759518 27972 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2: loading cluster ID for follower catalog manager: success
I20260620 01:51:21.762399 27972 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2: acquiring CA information for follower catalog manager: success
I20260620 01:51:21.765157 27972 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 01:51:21.777217 28000 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:21.777599 28000 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:21.777716 28000 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:21.787760 28000 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:21.788023 28000 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:51:21.800031 28000 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:21.802064 28000 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:21.804526 28000 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:21.817529 28006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:21.817477 28005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:21.817447 28008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:21.819590 28000 server_base.cc:1061] running on GCE node
I20260620 01:51:21.820559 28000 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:21.820775 28000 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:21.820889 28000 hybrid_clock.cc:648] HybridClock initialized: now 1781920281820865 us; error 0 us; skew 500 ppm
I20260620 01:51:21.821285 28000 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:21.824644 28000 webserver.cc:533] Webserver started at http://127.20.1.129:43981/ using document root <none> and password file <none>
I20260620 01:51:21.826195 28000 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:21.826407 28000 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:21.826977 28000 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:51:21.830579 28000 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "71bfcd49241b4427bf4049abf8a9b8f6"
format_stamp: "Formatted at 2026-06-20 01:51:21 on dist-test-slave-wg14"
I20260620 01:51:21.831585 28000 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "71bfcd49241b4427bf4049abf8a9b8f6"
format_stamp: "Formatted at 2026-06-20 01:51:21 on dist-test-slave-wg14"
I20260620 01:51:21.841128 28000 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260620 01:51:21.845809 28013 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:21.847990 28000 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 01:51:21.848263 28000 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "71bfcd49241b4427bf4049abf8a9b8f6"
format_stamp: "Formatted at 2026-06-20 01:51:21 on dist-test-slave-wg14"
I20260620 01:51:21.848526 28000 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:21.882483 28000 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:21.883715 28000 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:21.884076 28000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:21.885504 28000 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:51:21.887703 28000 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:51:21.887866 28000 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:21.887997 28000 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:51:21.888082 28000 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:21.914825 27986 catalog_manager.cc:1294] Loaded cluster ID: 18a4b0e44a1a4a1497f4c91cfd7df64b
I20260620 01:51:21.914985 27986 catalog_manager.cc:1587] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: loading cluster ID for follower catalog manager: success
I20260620 01:51:21.918608 27986 catalog_manager.cc:1609] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: acquiring CA information for follower catalog manager: success
I20260620 01:51:21.922456 27986 catalog_manager.cc:1637] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:51:21.933313 28000 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:36415
I20260620 01:51:21.933622 28125 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:36415 every 8 connection(s)
I20260620 01:51:21.934968 28000 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:51:21.943627 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 28000
I20260620 01:51:21.943849 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:51:21.952534 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--time_source=system_unsync with env {}
I20260620 01:51:21.978667 28126 heartbeater.cc:344] Connected to a master server at 127.20.1.188:43989
I20260620 01:51:21.979457 28126 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:21.981137 28126 heartbeater.cc:507] Master 127.20.1.188:43989 requested a full tablet report, sending...
I20260620 01:51:21.981176 28127 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44385
I20260620 01:51:21.981398 28127 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:21.982003 28127 heartbeater.cc:507] Master 127.20.1.190:44385 requested a full tablet report, sending...
I20260620 01:51:21.984624 27762 ts_manager.cc:194] Registered new tserver with Master: 71bfcd49241b4427bf4049abf8a9b8f6 (127.20.1.129:36415)
I20260620 01:51:21.984611 27903 ts_manager.cc:194] Registered new tserver with Master: 71bfcd49241b4427bf4049abf8a9b8f6 (127.20.1.129:36415)
I20260620 01:51:21.987289 27903 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:39561
I20260620 01:51:21.991017 28128 heartbeater.cc:344] Connected to a master server at 127.20.1.189:33769
I20260620 01:51:21.991289 28128 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:21.991741 28128 heartbeater.cc:507] Master 127.20.1.189:33769 requested a full tablet report, sending...
I20260620 01:51:21.994792 27832 ts_manager.cc:194] Registered new tserver with Master: 71bfcd49241b4427bf4049abf8a9b8f6 (127.20.1.129:36415)
W20260620 01:51:22.418825 28134 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:22.419263 28134 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:22.419384 28134 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:22.430595 28134 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:22.430851 28134 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:51:22.443574 28134 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:22.445964 28134 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:22.448380 28134 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:22.460836 28142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:22.460780 28141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:22.461479 28144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:22.462131 28134 server_base.cc:1061] running on GCE node
I20260620 01:51:22.462873 28134 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:22.463081 28134 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:22.463267 28134 hybrid_clock.cc:648] HybridClock initialized: now 1781920282463242 us; error 0 us; skew 500 ppm
I20260620 01:51:22.463603 28134 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:22.466603 28134 webserver.cc:533] Webserver started at http://127.20.1.130:39679/ using document root <none> and password file <none>
I20260620 01:51:22.467690 28134 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:22.467839 28134 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:22.468190 28134 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:51:22.471359 28134 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/data/instance:
uuid: "a0909a728b6f4d66953a00fdb76e8a7c"
format_stamp: "Formatted at 2026-06-20 01:51:22 on dist-test-slave-wg14"
I20260620 01:51:22.472528 28134 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance:
uuid: "a0909a728b6f4d66953a00fdb76e8a7c"
format_stamp: "Formatted at 2026-06-20 01:51:22 on dist-test-slave-wg14"
I20260620 01:51:22.480777 28134 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260620 01:51:22.486006 28149 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:22.488137 28134 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:51:22.488377 28134 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "a0909a728b6f4d66953a00fdb76e8a7c"
format_stamp: "Formatted at 2026-06-20 01:51:22 on dist-test-slave-wg14"
I20260620 01:51:22.488628 28134 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:22.511760 28134 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:22.513099 28134 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:22.513404 28134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:22.514750 28134 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:51:22.517545 28134 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:51:22.517771 28134 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:22.517895 28134 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:51:22.517970 28134 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:22.581907 28134 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:40427
I20260620 01:51:22.581961 28261 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:40427 every 8 connection(s)
I20260620 01:51:22.584671 28134 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:51:22.590358 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 28134
I20260620 01:51:22.590745 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
I20260620 01:51:22.596169 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--time_source=system_unsync with env {}
I20260620 01:51:22.629845 28262 heartbeater.cc:344] Connected to a master server at 127.20.1.188:43989
I20260620 01:51:22.629863 28263 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44385
I20260620 01:51:22.630462 28263 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:22.630462 28262 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:22.632629 28263 heartbeater.cc:507] Master 127.20.1.190:44385 requested a full tablet report, sending...
I20260620 01:51:22.632669 28262 heartbeater.cc:507] Master 127.20.1.188:43989 requested a full tablet report, sending...
I20260620 01:51:22.635995 27903 ts_manager.cc:194] Registered new tserver with Master: a0909a728b6f4d66953a00fdb76e8a7c (127.20.1.130:40427)
I20260620 01:51:22.636468 27762 ts_manager.cc:194] Registered new tserver with Master: a0909a728b6f4d66953a00fdb76e8a7c (127.20.1.130:40427)
I20260620 01:51:22.637411 27903 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.130:51287
I20260620 01:51:22.639647 28264 heartbeater.cc:344] Connected to a master server at 127.20.1.189:33769
I20260620 01:51:22.639889 28264 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:22.640393 28264 heartbeater.cc:507] Master 127.20.1.189:33769 requested a full tablet report, sending...
I20260620 01:51:22.642396 27832 ts_manager.cc:194] Registered new tserver with Master: a0909a728b6f4d66953a00fdb76e8a7c (127.20.1.130:40427)
I20260620 01:51:22.991603 28126 heartbeater.cc:499] Master 127.20.1.188:43989 was elected leader, sending a full tablet report...
W20260620 01:51:23.109455 28270 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:23.109930 28270 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:23.110518 28270 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:23.122129 28270 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:23.122409 28270 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:51:23.137292 28270 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:23.139477 28270 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:23.141908 28270 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:23.155447 28277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:23.155437 28278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:23.156085 28280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:23.157094 28270 server_base.cc:1061] running on GCE node
I20260620 01:51:23.158097 28270 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:23.158362 28270 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:23.158479 28270 hybrid_clock.cc:648] HybridClock initialized: now 1781920283158453 us; error 0 us; skew 500 ppm
I20260620 01:51:23.158842 28270 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:23.162293 28270 webserver.cc:533] Webserver started at http://127.20.1.131:36963/ using document root <none> and password file <none>
I20260620 01:51:23.163518 28270 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:23.163693 28270 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:23.164070 28270 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:51:23.169490 28270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/data/instance:
uuid: "32d43c43b0bd4da0bcea0f4c4a83ab7d"
format_stamp: "Formatted at 2026-06-20 01:51:23 on dist-test-slave-wg14"
I20260620 01:51:23.171607 28270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance:
uuid: "32d43c43b0bd4da0bcea0f4c4a83ab7d"
format_stamp: "Formatted at 2026-06-20 01:51:23 on dist-test-slave-wg14"
I20260620 01:51:23.179983 28270 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260620 01:51:23.185370 28285 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:23.187733 28270 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:51:23.188000 28270 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "32d43c43b0bd4da0bcea0f4c4a83ab7d"
format_stamp: "Formatted at 2026-06-20 01:51:23 on dist-test-slave-wg14"
I20260620 01:51:23.188536 28270 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:23.208130 28270 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:23.209525 28270 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:23.209875 28270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:23.211035 28270 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:51:23.213467 28270 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:51:23.213835 28270 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:23.214237 28270 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:51:23.214377 28270 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:23.271559 28270 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:41961
I20260620 01:51:23.271564 28397 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:41961 every 8 connection(s)
I20260620 01:51:23.273805 28270 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:51:23.275794 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 28270
I20260620 01:51:23.276036 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
I20260620 01:51:23.339466 28398 heartbeater.cc:344] Connected to a master server at 127.20.1.188:43989
I20260620 01:51:23.340063 28398 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:23.341820 28398 heartbeater.cc:507] Master 127.20.1.188:43989 requested a full tablet report, sending...
I20260620 01:51:23.343662 28399 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44385
I20260620 01:51:23.343838 28399 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:23.344226 27903 ts_manager.cc:194] Registered new tserver with Master: 32d43c43b0bd4da0bcea0f4c4a83ab7d (127.20.1.131:41961)
I20260620 01:51:23.344964 28399 heartbeater.cc:507] Master 127.20.1.190:44385 requested a full tablet report, sending...
I20260620 01:51:23.345701 27903 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.131:47383
I20260620 01:51:23.347922 27762 ts_manager.cc:194] Registered new tserver with Master: 32d43c43b0bd4da0bcea0f4c4a83ab7d (127.20.1.131:41961)
I20260620 01:51:23.353888 28400 heartbeater.cc:344] Connected to a master server at 127.20.1.189:33769
I20260620 01:51:23.354161 28400 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:23.355228 28400 heartbeater.cc:507] Master 127.20.1.189:33769 requested a full tablet report, sending...
I20260620 01:51:23.357438 27832 ts_manager.cc:194] Registered new tserver with Master: 32d43c43b0bd4da0bcea0f4c4a83ab7d (127.20.1.131:41961)
I20260620 01:51:23.373397 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:51:23.390256 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:51:23.390621 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 27732
I20260620 01:51:23.392470 27792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:23.537814 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 27732
I20260620 01:51:23.562513 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44385
--webserver_interface=127.20.1.190
--webserver_port=40955
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989 with env {}
I20260620 01:51:23.640637 28262 heartbeater.cc:499] Master 127.20.1.188:43989 was elected leader, sending a full tablet report...
W20260620 01:51:23.641482 28263 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:44385 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:44385: connect: Connection refused (error 111)
W20260620 01:51:23.808995 27888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.190:44385: connect: Connection refused (error 111)
W20260620 01:51:23.812158 27888 consensus_peers.cc:597] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 -> Peer e50045e01d1f47b9819454f38124309a (127.20.1.190:44385): Couldn't send request to peer e50045e01d1f47b9819454f38124309a. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:44385: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260620 01:51:23.992468 28127 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:44385 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:44385: connect: Connection refused (error 111)
W20260620 01:51:24.040442 28409 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:24.040853 28409 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:24.040977 28409 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:24.055437 28409 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:51:24.055603 28409 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:24.055706 28409 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:51:24.055774 28409 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:51:24.072147 28409 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=40955
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:24.076246 28409 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:24.078763 28409 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:24.090615 28418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:24.093456 28419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:24.097112 28421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:24.097935 28409 server_base.cc:1061] running on GCE node
I20260620 01:51:24.099459 28409 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:24.099709 28409 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:24.099833 28409 hybrid_clock.cc:648] HybridClock initialized: now 1781920284099805 us; error 0 us; skew 500 ppm
I20260620 01:51:24.100296 28409 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:24.106619 28409 webserver.cc:533] Webserver started at http://127.20.1.190:40955/ using document root <none> and password file <none>
I20260620 01:51:24.108608 28409 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:24.108827 28409 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:24.119592 28409 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 01:51:24.125880 28426 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:24.128400 28409 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.004s
I20260620 01:51:24.128688 28409 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "e50045e01d1f47b9819454f38124309a"
format_stamp: "Formatted at 2026-06-20 01:51:19 on dist-test-slave-wg14"
I20260620 01:51:24.129627 28409 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:24.171427 28409 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:24.172720 28409 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:24.173100 28409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:24.205314 28409 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44385
I20260620 01:51:24.205324 28477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44385 every 8 connection(s)
I20260620 01:51:24.207729 28409 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:51:24.215512 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 28409
I20260620 01:51:24.216131 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 28000
I20260620 01:51:24.219815 28478 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:51:24.224339 28478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: Bootstrap starting.
I20260620 01:51:24.235734 28121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:24.251595 28478 log.cc:826] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: Log is configured to *not* fsync() on all Append() calls
I20260620 01:51:24.266094 28478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:51:24.267328 28478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: Bootstrap complete.
I20260620 01:51:24.280146 28478 raft_consensus.cc:359] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:24.280987 28478 raft_consensus.cc:740] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50045e01d1f47b9819454f38124309a, State: Initialized, Role: FOLLOWER
I20260620 01:51:24.282552 28478 consensus_queue.cc:260] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:24.283805 28484 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:24.284286 28484 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:24.285149 28478 sys_catalog.cc:565] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:51:24.306885 28495 catalog_manager.cc:1294] Loaded cluster ID: 18a4b0e44a1a4a1497f4c91cfd7df64b
I20260620 01:51:24.307235 28495 catalog_manager.cc:1587] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: loading cluster ID for follower catalog manager: success
I20260620 01:51:24.318517 28495 catalog_manager.cc:1609] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: acquiring CA information for follower catalog manager: success
I20260620 01:51:24.331418 28495 catalog_manager.cc:1637] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:51:24.343626 27988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:24.348812 27988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: This master's current role is: LEADER
I20260620 01:51:24.350749 28398 heartbeater.cc:499] Master 127.20.1.188:43989 was elected leader, sending a full tablet report...
I20260620 01:51:24.365477 28484 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7acb459ab1bb42aab827fd0d715bc4a2. Latest consensus state: current_term: 1 leader_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:24.365870 28484 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:24.377287 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 28000
I20260620 01:51:24.384727 28443 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "32d43c43b0bd4da0bcea0f4c4a83ab7d" instance_seqno: 1781920283256877) as {username='slave'} at 127.20.1.131:48077; Asking this server to re-register.
I20260620 01:51:24.386412 28399 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:24.391731 28443 ts_manager.cc:194] Registered new tserver with Master: 32d43c43b0bd4da0bcea0f4c4a83ab7d (127.20.1.131:41961)
I20260620 01:51:24.409140 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:36415
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--webserver_port=43981
--webserver_interface=127.20.1.129
--time_source=system_unsync with env {}
I20260620 01:51:24.655764 28263 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44385
I20260620 01:51:24.657775 28443 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "a0909a728b6f4d66953a00fdb76e8a7c" instance_seqno: 1781920282560771) as {username='slave'} at 127.20.1.130:48233; Asking this server to re-register.
I20260620 01:51:24.658675 28263 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:24.660706 28443 ts_manager.cc:194] Registered new tserver with Master: a0909a728b6f4d66953a00fdb76e8a7c (127.20.1.130:40427)
I20260620 01:51:24.711922 27988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:24.712280 27988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: This master's current role is: LEADER
W20260620 01:51:24.911396 28506 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:24.911782 28506 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:24.911870 28506 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:24.922015 28506 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:24.922245 28506 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:51:24.934257 28506 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:36415
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=43981
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:24.936465 28506 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:24.938930 28506 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:24.953804 28513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:24.954617 28516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:24.954670 28514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:24.956285 28506 server_base.cc:1061] running on GCE node
I20260620 01:51:24.957337 28506 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:24.957733 28506 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:24.957901 28506 hybrid_clock.cc:648] HybridClock initialized: now 1781920284957873 us; error 0 us; skew 500 ppm
I20260620 01:51:24.958305 28506 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:24.962363 28506 webserver.cc:533] Webserver started at http://127.20.1.129:43981/ using document root <none> and password file <none>
I20260620 01:51:24.963599 28506 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:24.963773 28506 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:24.971556 28506 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 01:51:24.976979 28521 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:24.979434 28506 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:51:24.979717 28506 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "71bfcd49241b4427bf4049abf8a9b8f6"
format_stamp: "Formatted at 2026-06-20 01:51:21 on dist-test-slave-wg14"
I20260620 01:51:24.980633 28506 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:25.004411 28506 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:25.005726 28506 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:25.006062 28506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:25.007455 28506 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:51:25.009904 28506 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:51:25.010044 28506 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:25.010192 28506 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:51:25.010249 28506 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:25.056603 28506 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:36415
I20260620 01:51:25.056646 28633 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:36415 every 8 connection(s)
I20260620 01:51:25.059286 28506 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:51:25.068423 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 28506
I20260620 01:51:25.092345 28634 heartbeater.cc:344] Connected to a master server at 127.20.1.188:43989
I20260620 01:51:25.092980 28634 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:25.094412 28634 heartbeater.cc:507] Master 127.20.1.188:43989 requested a full tablet report, sending...
I20260620 01:51:25.096303 28635 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44385
I20260620 01:51:25.096586 28635 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:25.097393 28635 heartbeater.cc:507] Master 127.20.1.190:44385 requested a full tablet report, sending...
I20260620 01:51:25.098762 27903 ts_manager.cc:194] Re-registered known tserver with Master: 71bfcd49241b4427bf4049abf8a9b8f6 (127.20.1.129:36415)
I20260620 01:51:25.100746 27903 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:44225
I20260620 01:51:25.105432 28443 ts_manager.cc:194] Registered new tserver with Master: 71bfcd49241b4427bf4049abf8a9b8f6 (127.20.1.129:36415)
I20260620 01:51:25.110004 28636 heartbeater.cc:344] Connected to a master server at 127.20.1.189:33769
I20260620 01:51:25.110459 28636 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:25.111868 28636 heartbeater.cc:507] Master 127.20.1.189:33769 requested a full tablet report, sending...
I20260620 01:51:25.117337 27832 ts_manager.cc:194] Re-registered known tserver with Master: 71bfcd49241b4427bf4049abf8a9b8f6 (127.20.1.129:36415)
I20260620 01:51:25.138309 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 28506
I20260620 01:51:25.166667 28629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:25.273684 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 28506
I20260620 01:51:25.291343 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:36415
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--webserver_port=43981
--webserver_interface=127.20.1.129
--time_source=system_unsync
--fault_before_start=1.0 with env {}
W20260620 01:51:25.743696 28648 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:25.744050 28648 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:25.744148 28648 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:51:25.744211 28648 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:25.754899 28648 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:25.755227 28648 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:51:25.768064 28648 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:36415
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=43981
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:25.770107 28648 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:25.772413 28648 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:25.785969 28655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:25.786456 28656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:25.787654 28658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:25.788614 28648 server_base.cc:1061] running on GCE node
I20260620 01:51:25.789469 28648 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:25.789724 28648 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:25.789860 28648 hybrid_clock.cc:648] HybridClock initialized: now 1781920285789814 us; error 0 us; skew 500 ppm
I20260620 01:51:25.790282 28648 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:25.793900 28648 webserver.cc:533] Webserver started at http://127.20.1.129:43981/ using document root <none> and password file <none>
I20260620 01:51:25.795548 28648 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:25.795759 28648 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:25.804559 28648 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:51:25.808434 28663 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:25.811357 28648 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260620 01:51:25.811664 28648 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "71bfcd49241b4427bf4049abf8a9b8f6"
format_stamp: "Formatted at 2026-06-20 01:51:21 on dist-test-slave-wg14"
I20260620 01:51:25.812763 28648 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:25.843876 28648 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:25.845088 28648 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:25.845441 28648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:25.846978 28648 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:51:25.849170 28648 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:51:25.849310 28648 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:25.849452 28648 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:51:25.849532 28648 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:51:25.850065 28648 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:51:25.850211 28648 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:51:25.850291 28648 fault_injection.cc:59] injecting fault for kudu (pid 28648): FLAGS_fault_before_start (process will exit)
I20260620 01:51:25.871917 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:36415
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--webserver_port=43981
--webserver_interface=127.20.1.129
--time_source=system_unsync with env {}
W20260620 01:51:26.370242 28671 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:26.370687 28671 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:26.370862 28671 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:26.381575 28671 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:26.381806 28671 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:51:26.393551 28671 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:36415
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=43981
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:26.395493 28671 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:26.398062 28671 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:26.412770 28677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:26.412979 28676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:26.413003 28679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:26.414760 28671 server_base.cc:1061] running on GCE node
I20260620 01:51:26.416093 28671 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:26.416352 28671 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:26.416498 28671 hybrid_clock.cc:648] HybridClock initialized: now 1781920286416470 us; error 0 us; skew 500 ppm
I20260620 01:51:26.416914 28671 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:26.420161 28671 webserver.cc:533] Webserver started at http://127.20.1.129:43981/ using document root <none> and password file <none>
I20260620 01:51:26.421135 28671 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:26.421397 28671 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:26.429570 28671 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 01:51:26.436983 28684 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:26.439337 28671 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260620 01:51:26.439627 28671 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "71bfcd49241b4427bf4049abf8a9b8f6"
format_stamp: "Formatted at 2026-06-20 01:51:21 on dist-test-slave-wg14"
I20260620 01:51:26.440743 28671 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:26.470968 28671 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:26.472460 28671 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:26.472801 28671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:26.474184 28671 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:51:26.476876 28671 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:51:26.477000 28671 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:26.477180 28671 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:51:26.477241 28671 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:26.520632 28796 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:36415 every 8 connection(s)
I20260620 01:51:26.520656 28671 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:36415
I20260620 01:51:26.523767 28671 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:51:26.533859 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 28671
I20260620 01:51:26.534267 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 28671
I20260620 01:51:26.572606 28797 heartbeater.cc:344] Connected to a master server at 127.20.1.188:43989
I20260620 01:51:26.572844 28799 heartbeater.cc:344] Connected to a master server at 127.20.1.189:33769
I20260620 01:51:26.573228 28797 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:26.573237 28799 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:26.575022 28792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:26.575194 28797 heartbeater.cc:507] Master 127.20.1.188:43989 requested a full tablet report, sending...
I20260620 01:51:26.676752 28798 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44385
I20260620 01:51:26.677436 28799 heartbeater.cc:507] Master 127.20.1.189:33769 requested a full tablet report, sending...
I20260620 01:51:26.677558 27903 ts_manager.cc:194] Re-registered known tserver with Master: 71bfcd49241b4427bf4049abf8a9b8f6 (127.20.1.129:36415)
I20260620 01:51:26.678632 28798 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:26.678853 27903 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:47293
I20260620 01:51:26.679409 27832 ts_manager.cc:194] Re-registered known tserver with Master: 71bfcd49241b4427bf4049abf8a9b8f6 (127.20.1.129:36415)
I20260620 01:51:26.679538 28798 heartbeater.cc:507] Master 127.20.1.190:44385 requested a full tablet report, sending...
I20260620 01:51:26.680375 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 28671
I20260620 01:51:26.681321 28443 ts_manager.cc:194] Re-registered known tserver with Master: 71bfcd49241b4427bf4049abf8a9b8f6 (127.20.1.129:36415)
W20260620 01:51:26.697009 28430 connection.cc:573] server connection from 127.20.1.129:52895 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:51:26.697440 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 28134
I20260620 01:51:26.708600 28257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:26.814291 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 28134
I20260620 01:51:26.829633 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 28270
I20260620 01:51:26.841965 28393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:26.953816 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 28270
I20260620 01:51:26.970116 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 28409
I20260620 01:51:26.979419 28473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:27.105268 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 28409
I20260620 01:51:27.130174 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 27799
I20260620 01:51:27.131637 27862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:27.261672 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 27799
I20260620 01:51:27.281014 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 27869
I20260620 01:51:27.282539 27933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:27.425766 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 27869
Shutting down hive metastore.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/security.properties
2026-06-20 01:51:29: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260620 01:51:39.606112 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44385
--webserver_interface=127.20.1.190
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989 with env {}
W20260620 01:51:40.062610 29001 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:40.063050 29001 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:40.063212 29001 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:40.077883 29001 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:51:40.078052 29001 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:40.078145 29001 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:51:40.078223 29001 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:51:40.092803 29001 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:40.094904 29001 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:40.097962 29001 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:40.122773 29006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:40.122737 29007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:40.129112 29009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:40.129920 29001 server_base.cc:1061] running on GCE node
I20260620 01:51:40.131572 29001 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:40.131901 29001 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:40.132014 29001 hybrid_clock.cc:648] HybridClock initialized: now 1781920300131984 us; error 0 us; skew 500 ppm
I20260620 01:51:40.132476 29001 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:40.137342 29001 webserver.cc:533] Webserver started at http://127.20.1.190:43381/ using document root <none> and password file <none>
I20260620 01:51:40.139372 29001 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:40.139853 29001 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:40.150254 29001 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 01:51:40.154863 29014 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:40.157387 29001 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:51:40.157675 29001 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "e50045e01d1f47b9819454f38124309a"
format_stamp: "Formatted at 2026-06-20 01:51:19 on dist-test-slave-wg14"
I20260620 01:51:40.158636 29001 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:40.184015 29001 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:40.185992 29001 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:40.186489 29001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:40.216264 29001 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44385
I20260620 01:51:40.216367 29065 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44385 every 8 connection(s)
I20260620 01:51:40.218922 29001 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:51:40.221387 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 29001
I20260620 01:51:40.221624 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:51:40.223174 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:33769
--webserver_interface=127.20.1.189
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989 with env {}
I20260620 01:51:40.232007 29066 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:51:40.236824 29066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: Bootstrap starting.
I20260620 01:51:40.261468 29066 log.cc:826] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: Log is configured to *not* fsync() on all Append() calls
I20260620 01:51:40.277874 29066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:51:40.278635 29066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: Bootstrap complete.
I20260620 01:51:40.287704 29066 raft_consensus.cc:359] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:40.289518 29066 raft_consensus.cc:740] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50045e01d1f47b9819454f38124309a, State: Initialized, Role: FOLLOWER
I20260620 01:51:40.291522 29066 consensus_queue.cc:260] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:40.294028 29070 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:40.294826 29070 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:40.296568 29066 sys_catalog.cc:565] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:51:40.322028 29081 catalog_manager.cc:1294] Loaded cluster ID: 18a4b0e44a1a4a1497f4c91cfd7df64b
I20260620 01:51:40.322500 29081 catalog_manager.cc:1587] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: loading cluster ID for follower catalog manager: success
I20260620 01:51:40.331903 29081 catalog_manager.cc:1609] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: acquiring CA information for follower catalog manager: success
I20260620 01:51:40.337663 29081 catalog_manager.cc:1637] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:51:40.352871 29001 master_runner.cc:187] Error getting master registration for 127.20.1.189:33769: Network error: Client connection negotiation failed: client connection to 127.20.1.189:33769: connect: Connection refused (error 111),
I20260620 01:51:40.361441 29001 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:51:40.721282 29068 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:40.721855 29068 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:40.721980 29068 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:40.733714 29068 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:51:40.733872 29068 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:40.733961 29068 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:51:40.734026 29068 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:51:40.748445 29068 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:33769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:40.750538 29068 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:40.753129 29068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:40.765537 29089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:40.766417 29090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:40.767645 29092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:40.767738 29068 server_base.cc:1061] running on GCE node
I20260620 01:51:40.769035 29068 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:40.769328 29068 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:40.769457 29068 hybrid_clock.cc:648] HybridClock initialized: now 1781920300769428 us; error 0 us; skew 500 ppm
I20260620 01:51:40.770008 29068 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:40.777097 29068 webserver.cc:533] Webserver started at http://127.20.1.189:37763/ using document root <none> and password file <none>
I20260620 01:51:40.778589 29068 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:40.778832 29068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:40.790256 29068 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.007s sys 0.001s
I20260620 01:51:40.796954 29097 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:40.799662 29068 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.002s
I20260620 01:51:40.800005 29068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "324f486c9a9043f2b98e035d3d5528d2"
format_stamp: "Formatted at 2026-06-20 01:51:19 on dist-test-slave-wg14"
I20260620 01:51:40.801929 29068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:40.842954 29068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:40.844592 29068 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:40.845314 29068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:40.879016 29068 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:33769
I20260620 01:51:40.879056 29148 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:33769 every 8 connection(s)
I20260620 01:51:40.882061 29068 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:51:40.889557 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 29068
I20260620 01:51:40.889832 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:51:40.892038 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:43989
--webserver_interface=127.20.1.188
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989 with env {}
I20260620 01:51:40.894804 29149 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:51:40.899446 29149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2: Bootstrap starting.
I20260620 01:51:40.937409 29149 log.cc:826] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2: Log is configured to *not* fsync() on all Append() calls
I20260620 01:51:40.951901 29149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:51:40.952605 29149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2: Bootstrap complete.
I20260620 01:51:40.961748 29149 raft_consensus.cc:359] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:40.962868 29149 raft_consensus.cc:740] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324f486c9a9043f2b98e035d3d5528d2, State: Initialized, Role: FOLLOWER
I20260620 01:51:40.964320 29149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:40.965961 29153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:40.966571 29153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:40.967768 29149 sys_catalog.cc:565] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:51:41.000891 29164 catalog_manager.cc:1294] Loaded cluster ID: 18a4b0e44a1a4a1497f4c91cfd7df64b
I20260620 01:51:41.001060 29164 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2: loading cluster ID for follower catalog manager: success
I20260620 01:51:41.006287 29164 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2: acquiring CA information for follower catalog manager: success
I20260620 01:51:41.011826 29164 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:51:41.049494 29068 master_runner.cc:187] Error getting master registration for 127.20.1.188:43989: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43989: connect: Connection refused (error 111),
I20260620 01:51:41.058970 29068 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:51:41.461766 29151 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:41.462193 29151 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:41.462307 29151 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:41.479453 29151 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:51:41.479585 29151 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:41.479661 29151 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:51:41.479738 29151 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:51:41.496991 29151 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:43989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:41.499282 29151 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:41.502112 29151 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:41.516011 29173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:41.516117 29174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:41.519117 29176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:41.519685 29151 server_base.cc:1061] running on GCE node
I20260620 01:51:41.520889 29151 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:41.521257 29151 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:41.521500 29151 hybrid_clock.cc:648] HybridClock initialized: now 1781920301521466 us; error 0 us; skew 500 ppm
I20260620 01:51:41.522044 29151 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:41.527002 29151 webserver.cc:533] Webserver started at http://127.20.1.188:42319/ using document root <none> and password file <none>
I20260620 01:51:41.528143 29151 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:41.528334 29151 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:41.537325 29151 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:51:41.541812 29181 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:41.544431 29151 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:51:41.544710 29151 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "7acb459ab1bb42aab827fd0d715bc4a2"
format_stamp: "Formatted at 2026-06-20 01:51:20 on dist-test-slave-wg14"
I20260620 01:51:41.545919 29151 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:41.605567 29151 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:41.607293 29151 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:41.607950 29151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:41.637027 29151 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:43989
I20260620 01:51:41.637070 29232 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:43989 every 8 connection(s)
I20260620 01:51:41.640065 29151 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:51:41.649194 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 29151
I20260620 01:51:41.649394 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:51:41.650066 29233 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:51:41.655879 29233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2: Bootstrap starting.
I20260620 01:51:41.682513 29233 log.cc:826] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2: Log is configured to *not* fsync() on all Append() calls
I20260620 01:51:41.696527 29233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:51:41.697134 29233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2: Bootstrap complete.
I20260620 01:51:41.706844 29233 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:41.708415 29233 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7acb459ab1bb42aab827fd0d715bc4a2, State: Initialized, Role: FOLLOWER
I20260620 01:51:41.709983 29233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:41.711462 29240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:41.711794 29240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:41.713898 29233 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:51:41.737576 29251 catalog_manager.cc:1294] Loaded cluster ID: 18a4b0e44a1a4a1497f4c91cfd7df64b
I20260620 01:51:41.737805 29251 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2: loading cluster ID for follower catalog manager: success
I20260620 01:51:41.743906 29251 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2: acquiring CA information for follower catalog manager: success
I20260620 01:51:41.752107 29251 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:51:41.779413 29151 master_runner.cc:264] No leader master found from master 7acb459ab1bb42aab827fd0d715bc4a2
I20260620 01:51:41.779620 29151 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:51:41.989832 29254 raft_consensus.cc:493] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:51:41.990275 29254 raft_consensus.cc:515] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:41.994913 29254 leader_election.cc:290] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 324f486c9a9043f2b98e035d3d5528d2 (127.20.1.189:33769), 7acb459ab1bb42aab827fd0d715bc4a2 (127.20.1.188:43989)
I20260620 01:51:42.004662 29124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e50045e01d1f47b9819454f38124309a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "324f486c9a9043f2b98e035d3d5528d2" is_pre_election: true
I20260620 01:51:42.005362 29124 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e50045e01d1f47b9819454f38124309a in term 1.
I20260620 01:51:42.006168 29016 leader_election.cc:304] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 324f486c9a9043f2b98e035d3d5528d2, e50045e01d1f47b9819454f38124309a; no voters:
I20260620 01:51:42.006779 29254 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:51:42.006999 29254 raft_consensus.cc:493] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:51:42.007674 29254 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:51:42.010052 29208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e50045e01d1f47b9819454f38124309a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" is_pre_election: true
I20260620 01:51:42.010888 29208 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e50045e01d1f47b9819454f38124309a in term 1.
I20260620 01:51:42.015094 29254 raft_consensus.cc:515] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:42.016589 29254 leader_election.cc:290] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [CANDIDATE]: Term 2 election: Requested vote from peers 324f486c9a9043f2b98e035d3d5528d2 (127.20.1.189:33769), 7acb459ab1bb42aab827fd0d715bc4a2 (127.20.1.188:43989)
I20260620 01:51:42.017366 29124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e50045e01d1f47b9819454f38124309a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "324f486c9a9043f2b98e035d3d5528d2"
I20260620 01:51:42.017685 29124 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:51:42.017714 29208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e50045e01d1f47b9819454f38124309a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7acb459ab1bb42aab827fd0d715bc4a2"
I20260620 01:51:42.017975 29208 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:51:42.024158 29124 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e50045e01d1f47b9819454f38124309a in term 2.
I20260620 01:51:42.026536 29016 leader_election.cc:304] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 324f486c9a9043f2b98e035d3d5528d2, e50045e01d1f47b9819454f38124309a; no voters:
I20260620 01:51:42.027697 29254 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 01:51:42.028890 29208 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e50045e01d1f47b9819454f38124309a in term 2.
I20260620 01:51:42.029546 29254 raft_consensus.cc:697] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 2 LEADER]: Becoming Leader. State: Replica: e50045e01d1f47b9819454f38124309a, State: Running, Role: LEADER
I20260620 01:51:42.031317 29254 consensus_queue.cc:237] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:42.037662 29257 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e50045e01d1f47b9819454f38124309a. Latest consensus state: current_term: 2 leader_uuid: "e50045e01d1f47b9819454f38124309a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:42.038117 29257 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: This master's current role is: LEADER
I20260620 01:51:42.039225 29259 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:51:42.041751 29259 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:51:42.042971 29259 catalog_manager.cc:1294] Loaded cluster ID: 18a4b0e44a1a4a1497f4c91cfd7df64b
I20260620 01:51:42.043114 29259 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:51:42.044593 29259 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:51:42.045573 29259 catalog_manager.cc:6080] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: Loaded TSK: 0
I20260620 01:51:42.046738 29259 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:51:42.059868 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--time_source=system_unsync with env {}
W20260620 01:51:42.564640 29260 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:42.565064 29260 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:42.565158 29260 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:42.575094 29260 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:42.575374 29260 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:51:42.582971 29124 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 2 FOLLOWER]: Refusing update from remote peer e50045e01d1f47b9819454f38124309a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:51:42.584861 29264 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [LEADER]: Connected to new peer: Peer: permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:51:42.590421 29260 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:42.590354 29265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e50045e01d1f47b9819454f38124309a. Latest consensus state: current_term: 2 leader_uuid: "e50045e01d1f47b9819454f38124309a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:42.590798 29265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:42.593146 29260 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:42.593865 29264 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e50045e01d1f47b9819454f38124309a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:42.594515 29264 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: This master's current role is: LEADER
I20260620 01:51:42.597295 29208 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 2 FOLLOWER]: Refusing update from remote peer e50045e01d1f47b9819454f38124309a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:51:42.598073 29260 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260620 01:51:42.608273 29264 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:51:42.611109 29265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e50045e01d1f47b9819454f38124309a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:42.611550 29265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:42.616364 29277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e50045e01d1f47b9819454f38124309a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:42.616662 29272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e50045e01d1f47b9819454f38124309a. Latest consensus state: current_term: 2 leader_uuid: "e50045e01d1f47b9819454f38124309a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:42.616851 29277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:42.616906 29272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:51:42.619417 29275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:42.620137 29278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:42.623109 29280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:42.623718 29260 server_base.cc:1061] running on GCE node
I20260620 01:51:42.624522 29260 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:42.624761 29260 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:42.624890 29260 hybrid_clock.cc:648] HybridClock initialized: now 1781920302624866 us; error 0 us; skew 500 ppm
I20260620 01:51:42.625293 29260 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:42.630739 29260 webserver.cc:533] Webserver started at http://127.20.1.129:43245/ using document root <none> and password file <none>
I20260620 01:51:42.632622 29260 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:42.632827 29260 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:42.643450 29260 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.003s sys 0.004s
I20260620 01:51:42.648751 29285 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:42.651742 29260 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:51:42.652298 29260 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "71bfcd49241b4427bf4049abf8a9b8f6"
format_stamp: "Formatted at 2026-06-20 01:51:21 on dist-test-slave-wg14"
I20260620 01:51:42.654078 29260 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:42.676498 29260 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:42.677922 29260 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:42.678274 29260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:42.679991 29260 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:51:42.682943 29260 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:51:42.683076 29260 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:42.683243 29260 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:51:42.683306 29260 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:42.730651 29260 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:46781
I20260620 01:51:42.730736 29397 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:46781 every 8 connection(s)
I20260620 01:51:42.733783 29260 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:51:42.734018 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 29260
I20260620 01:51:42.734262 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:51:42.740335 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--time_source=system_unsync with env {}
I20260620 01:51:42.783965 29399 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44385
I20260620 01:51:42.784015 29398 heartbeater.cc:344] Connected to a master server at 127.20.1.188:43989
I20260620 01:51:42.784600 29398 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:42.784605 29399 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:42.786413 29398 heartbeater.cc:507] Master 127.20.1.188:43989 requested a full tablet report, sending...
I20260620 01:51:42.786425 29399 heartbeater.cc:507] Master 127.20.1.190:44385 requested a full tablet report, sending...
I20260620 01:51:42.790557 29031 ts_manager.cc:194] Registered new tserver with Master: 71bfcd49241b4427bf4049abf8a9b8f6 (127.20.1.129:46781)
I20260620 01:51:42.790746 29198 ts_manager.cc:194] Registered new tserver with Master: 71bfcd49241b4427bf4049abf8a9b8f6 (127.20.1.129:46781)
I20260620 01:51:42.794010 29031 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:38155
I20260620 01:51:42.797646 29400 heartbeater.cc:344] Connected to a master server at 127.20.1.189:33769
I20260620 01:51:42.797915 29400 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:42.798358 29400 heartbeater.cc:507] Master 127.20.1.189:33769 requested a full tablet report, sending...
I20260620 01:51:42.800930 29114 ts_manager.cc:194] Registered new tserver with Master: 71bfcd49241b4427bf4049abf8a9b8f6 (127.20.1.129:46781)
W20260620 01:51:43.210734 29406 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:43.211256 29406 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:43.211426 29406 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:43.227924 29406 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:43.228470 29406 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:51:43.229480 29414 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e50045e01d1f47b9819454f38124309a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:43.229925 29414 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: This master's current role is: LEADER
I20260620 01:51:43.240823 29406 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:43.242848 29406 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:43.245180 29406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:43.259378 29418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:43.259505 29417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:43.259394 29420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:43.260833 29406 server_base.cc:1061] running on GCE node
I20260620 01:51:43.261834 29406 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:43.262087 29406 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:43.262208 29406 hybrid_clock.cc:648] HybridClock initialized: now 1781920303262183 us; error 0 us; skew 500 ppm
I20260620 01:51:43.262593 29406 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:43.265830 29406 webserver.cc:533] Webserver started at http://127.20.1.130:36825/ using document root <none> and password file <none>
I20260620 01:51:43.267253 29406 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:43.267432 29406 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:43.275076 29406 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 01:51:43.279321 29425 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:43.281828 29406 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:51:43.282075 29406 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "a0909a728b6f4d66953a00fdb76e8a7c"
format_stamp: "Formatted at 2026-06-20 01:51:22 on dist-test-slave-wg14"
I20260620 01:51:43.283010 29406 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:43.301812 29406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:43.302910 29406 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:43.303293 29406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:43.304755 29406 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:51:43.307725 29406 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:51:43.307906 29406 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:43.308053 29406 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:51:43.308117 29406 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:43.352291 29406 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:39065
I20260620 01:51:43.352349 29537 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:39065 every 8 connection(s)
I20260620 01:51:43.354544 29406 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:51:43.357280 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 29406
I20260620 01:51:43.357599 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
I20260620 01:51:43.366636 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--time_source=system_unsync with env {}
I20260620 01:51:43.391386 29538 heartbeater.cc:344] Connected to a master server at 127.20.1.188:43989
I20260620 01:51:43.391315 29540 heartbeater.cc:344] Connected to a master server at 127.20.1.189:33769
I20260620 01:51:43.391954 29540 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:43.392393 29538 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:43.393673 29538 heartbeater.cc:507] Master 127.20.1.188:43989 requested a full tablet report, sending...
I20260620 01:51:43.393704 29540 heartbeater.cc:507] Master 127.20.1.189:33769 requested a full tablet report, sending...
I20260620 01:51:43.396080 29539 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44385
I20260620 01:51:43.396265 29539 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:43.396222 29114 ts_manager.cc:194] Registered new tserver with Master: a0909a728b6f4d66953a00fdb76e8a7c (127.20.1.130:39065)
I20260620 01:51:43.396227 29198 ts_manager.cc:194] Registered new tserver with Master: a0909a728b6f4d66953a00fdb76e8a7c (127.20.1.130:39065)
I20260620 01:51:43.396943 29539 heartbeater.cc:507] Master 127.20.1.190:44385 requested a full tablet report, sending...
I20260620 01:51:43.398581 29031 ts_manager.cc:194] Registered new tserver with Master: a0909a728b6f4d66953a00fdb76e8a7c (127.20.1.130:39065)
I20260620 01:51:43.399657 29031 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.130:60735
I20260620 01:51:43.798311 29399 heartbeater.cc:499] Master 127.20.1.190:44385 was elected leader, sending a full tablet report...
W20260620 01:51:43.812228 29547 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:43.812821 29547 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:43.813058 29547 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:43.825296 29547 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:43.825613 29547 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:51:43.837607 29547 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:43.839811 29547 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:43.842140 29547 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:43.854895 29554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:43.855938 29553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:43.856259 29556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:43.857986 29547 server_base.cc:1061] running on GCE node
I20260620 01:51:43.858945 29547 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:43.859287 29547 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:43.859427 29547 hybrid_clock.cc:648] HybridClock initialized: now 1781920303859403 us; error 0 us; skew 500 ppm
I20260620 01:51:43.859786 29547 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:43.862944 29547 webserver.cc:533] Webserver started at http://127.20.1.131:45713/ using document root <none> and password file <none>
I20260620 01:51:43.864004 29547 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:43.864194 29547 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:43.872964 29547 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 01:51:43.877472 29561 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:43.880761 29547 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260620 01:51:43.881093 29547 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "32d43c43b0bd4da0bcea0f4c4a83ab7d"
format_stamp: "Formatted at 2026-06-20 01:51:23 on dist-test-slave-wg14"
I20260620 01:51:43.882519 29547 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:43.921140 29547 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:43.922452 29547 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:43.922787 29547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:43.924144 29547 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:51:43.926784 29547 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:51:43.926911 29547 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:43.927069 29547 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:51:43.927178 29547 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:43.975967 29547 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:42321
I20260620 01:51:43.976109 29673 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:42321 every 8 connection(s)
I20260620 01:51:43.978408 29547 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:51:43.987902 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 29547
I20260620 01:51:43.988555 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
I20260620 01:51:44.021096 29675 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44385
I20260620 01:51:44.021256 29676 heartbeater.cc:344] Connected to a master server at 127.20.1.189:33769
I20260620 01:51:44.021636 29676 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:44.021644 29675 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:44.023063 29674 heartbeater.cc:344] Connected to a master server at 127.20.1.188:43989
I20260620 01:51:44.023305 29674 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:44.023497 29676 heartbeater.cc:507] Master 127.20.1.189:33769 requested a full tablet report, sending...
I20260620 01:51:44.023528 29675 heartbeater.cc:507] Master 127.20.1.190:44385 requested a full tablet report, sending...
I20260620 01:51:44.023972 29674 heartbeater.cc:507] Master 127.20.1.188:43989 requested a full tablet report, sending...
I20260620 01:51:44.026083 29198 ts_manager.cc:194] Registered new tserver with Master: 32d43c43b0bd4da0bcea0f4c4a83ab7d (127.20.1.131:42321)
I20260620 01:51:44.026083 29114 ts_manager.cc:194] Registered new tserver with Master: 32d43c43b0bd4da0bcea0f4c4a83ab7d (127.20.1.131:42321)
I20260620 01:51:44.026937 29031 ts_manager.cc:194] Registered new tserver with Master: 32d43c43b0bd4da0bcea0f4c4a83ab7d (127.20.1.131:42321)
I20260620 01:51:44.027961 29031 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.131:44589
I20260620 01:51:44.039803 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:51:44.054026 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:51:44.054306 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29001
I20260620 01:51:44.055809 29061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:44.190292 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 29001
W20260620 01:51:44.213200 29184 connection.cc:573] server connection from 127.0.0.1:47186 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:51:44.214315 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44385
--webserver_interface=127.20.1.190
--webserver_port=43381
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989 with env {}
I20260620 01:51:44.402937 29539 heartbeater.cc:499] Master 127.20.1.190:44385 was elected leader, sending a full tablet report...
W20260620 01:51:44.405133 29539 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:44385 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:44385: connect: Connection refused (error 111)
W20260620 01:51:44.725477 29686 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:44.725894 29686 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:44.726001 29686 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:44.737378 29686 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:51:44.737519 29686 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:44.737584 29686 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:51:44.737660 29686 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:51:44.749531 29686 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=43381
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:44.751317 29686 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:44.753661 29686 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:44.765255 29693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:44.765218 29692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:44.765569 29695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:44.766700 29686 server_base.cc:1061] running on GCE node
I20260620 01:51:44.768384 29686 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:44.768625 29686 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:44.768718 29686 hybrid_clock.cc:648] HybridClock initialized: now 1781920304768696 us; error 0 us; skew 500 ppm
I20260620 01:51:44.769153 29686 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:44.773108 29686 webserver.cc:533] Webserver started at http://127.20.1.190:43381/ using document root <none> and password file <none>
I20260620 01:51:44.774159 29686 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:44.774322 29686 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:44.783388 29686 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.003s sys 0.004s
I20260620 01:51:44.788913 29700 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:44.791967 29686 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260620 01:51:44.792335 29686 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "e50045e01d1f47b9819454f38124309a"
format_stamp: "Formatted at 2026-06-20 01:51:19 on dist-test-slave-wg14"
I20260620 01:51:44.793875 29686 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260620 01:51:44.804629 29399 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:44385 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:44385: connect: Connection refused (error 111)
I20260620 01:51:44.821712 29686 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:44.822880 29686 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:44.823290 29686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:44.850119 29686 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44385
I20260620 01:51:44.850160 29752 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44385 every 8 connection(s)
I20260620 01:51:44.852167 29686 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:51:44.859963 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 29686
I20260620 01:51:44.860407 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29260
I20260620 01:51:44.862949 29753 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:51:44.868607 29753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: Bootstrap starting.
I20260620 01:51:44.875953 29393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:44.896044 29753 log.cc:826] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: Log is configured to *not* fsync() on all Append() calls
I20260620 01:51:44.912722 29753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:51:44.913551 29753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: Bootstrap complete.
I20260620 01:51:44.922989 29753 raft_consensus.cc:359] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:44.923899 29753 raft_consensus.cc:740] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50045e01d1f47b9819454f38124309a, State: Initialized, Role: FOLLOWER
I20260620 01:51:44.924968 29753 consensus_queue.cc:260] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:44.926259 29759 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:44.926663 29759 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:44.927851 29753 sys_catalog.cc:565] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:51:44.954679 29770 catalog_manager.cc:1294] Loaded cluster ID: 18a4b0e44a1a4a1497f4c91cfd7df64b
I20260620 01:51:44.954957 29770 catalog_manager.cc:1587] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: loading cluster ID for follower catalog manager: success
I20260620 01:51:44.960533 29770 catalog_manager.cc:1609] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: acquiring CA information for follower catalog manager: success
I20260620 01:51:44.971493 29770 catalog_manager.cc:1637] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:51:45.021363 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 29260
I20260620 01:51:45.028867 29686 master_runner.cc:264] No leader master found from master e50045e01d1f47b9819454f38124309a
I20260620 01:51:45.029079 29686 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:51:45.032122 29675 heartbeater.cc:499] Master 127.20.1.190:44385 was elected leader, sending a full tablet report...
I20260620 01:51:45.041908 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:46781
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--webserver_port=43245
--webserver_interface=127.20.1.129
--time_source=system_unsync with env {}
I20260620 01:51:45.046070 29718 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "32d43c43b0bd4da0bcea0f4c4a83ab7d" instance_seqno: 1781920303963370) as {username='slave'} at 127.20.1.131:46013; Asking this server to re-register.
I20260620 01:51:45.047375 29675 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:45.049605 29718 ts_manager.cc:194] Registered new tserver with Master: 32d43c43b0bd4da0bcea0f4c4a83ab7d (127.20.1.131:42321)
I20260620 01:51:45.419570 29539 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44385
I20260620 01:51:45.419948 29539 heartbeater.cc:499] Master 127.20.1.190:44385 was elected leader, sending a full tablet report...
I20260620 01:51:45.421588 29718 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "a0909a728b6f4d66953a00fdb76e8a7c" instance_seqno: 1781920303341998) as {username='slave'} at 127.20.1.130:37401; Asking this server to re-register.
I20260620 01:51:45.422860 29539 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:45.425236 29718 ts_manager.cc:194] Registered new tserver with Master: a0909a728b6f4d66953a00fdb76e8a7c (127.20.1.130:39065)
W20260620 01:51:45.555655 29777 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:45.556152 29777 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:45.556279 29777 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260620 01:51:45.558631 29782 raft_consensus.cc:493] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e50045e01d1f47b9819454f38124309a)
I20260620 01:51:45.558917 29782 raft_consensus.cc:515] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:45.561450 29782 leader_election.cc:290] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e50045e01d1f47b9819454f38124309a (127.20.1.190:44385), 7acb459ab1bb42aab827fd0d715bc4a2 (127.20.1.188:43989)
W20260620 01:51:45.570715 29777 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:45.571063 29777 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:51:45.573930 29208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "324f486c9a9043f2b98e035d3d5528d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" is_pre_election: true
I20260620 01:51:45.575093 29728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "324f486c9a9043f2b98e035d3d5528d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e50045e01d1f47b9819454f38124309a" is_pre_election: true
I20260620 01:51:45.575691 29728 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 324f486c9a9043f2b98e035d3d5528d2 in term 2.
I20260620 01:51:45.576418 29099 leader_election.cc:304] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 324f486c9a9043f2b98e035d3d5528d2, e50045e01d1f47b9819454f38124309a; no voters: 7acb459ab1bb42aab827fd0d715bc4a2
I20260620 01:51:45.577186 29782 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 01:51:45.577396 29782 raft_consensus.cc:493] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e50045e01d1f47b9819454f38124309a)
I20260620 01:51:45.577790 29782 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:51:45.580173 29782 raft_consensus.cc:515] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:45.580999 29782 leader_election.cc:290] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [CANDIDATE]: Term 3 election: Requested vote from peers e50045e01d1f47b9819454f38124309a (127.20.1.190:44385), 7acb459ab1bb42aab827fd0d715bc4a2 (127.20.1.188:43989)
I20260620 01:51:45.581666 29208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "324f486c9a9043f2b98e035d3d5528d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7acb459ab1bb42aab827fd0d715bc4a2"
I20260620 01:51:45.581688 29728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "324f486c9a9043f2b98e035d3d5528d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e50045e01d1f47b9819454f38124309a"
I20260620 01:51:45.581933 29728 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:51:45.586519 29728 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 324f486c9a9043f2b98e035d3d5528d2 in term 3.
I20260620 01:51:45.586527 29777 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:46781
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=43245
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:45.587440 29099 leader_election.cc:304] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 324f486c9a9043f2b98e035d3d5528d2, e50045e01d1f47b9819454f38124309a; no voters: 7acb459ab1bb42aab827fd0d715bc4a2
I20260620 01:51:45.587877 29782 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 01:51:45.588439 29782 raft_consensus.cc:697] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [term 3 LEADER]: Becoming Leader. State: Replica: 324f486c9a9043f2b98e035d3d5528d2, State: Running, Role: LEADER
I20260620 01:51:45.589051 29782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:45.590493 29777 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:45.591495 29786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 324f486c9a9043f2b98e035d3d5528d2. Latest consensus state: current_term: 3 leader_uuid: "324f486c9a9043f2b98e035d3d5528d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:45.591955 29786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: This master's current role is: LEADER
I20260620 01:51:45.592875 29788 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:51:45.593976 29777 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260620 01:51:45.595643 29788 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:51:45.596735 29788 catalog_manager.cc:1294] Loaded cluster ID: 18a4b0e44a1a4a1497f4c91cfd7df64b
I20260620 01:51:45.596859 29788 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:51:45.599036 29788 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:51:45.600461 29788 catalog_manager.cc:6080] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2: Loaded TSK: 0
I20260620 01:51:45.601809 29788 catalog_manager.cc:1549] Initializing in-progress tserver states...
W20260620 01:51:45.608404 29791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:45.608446 29793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:45.608383 29790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:45.610090 29777 server_base.cc:1061] running on GCE node
I20260620 01:51:45.610935 29777 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:45.611183 29777 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:45.611296 29777 hybrid_clock.cc:648] HybridClock initialized: now 1781920305611267 us; error 0 us; skew 500 ppm
I20260620 01:51:45.611677 29777 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:45.615316 29777 webserver.cc:533] Webserver started at http://127.20.1.129:43245/ using document root <none> and password file <none>
I20260620 01:51:45.616344 29777 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:45.616482 29777 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:45.625145 29777 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 01:51:45.629791 29798 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:45.632339 29777 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:51:45.632620 29777 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "71bfcd49241b4427bf4049abf8a9b8f6"
format_stamp: "Formatted at 2026-06-20 01:51:21 on dist-test-slave-wg14"
I20260620 01:51:45.633569 29777 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:45.668876 29777 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:45.669996 29777 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:45.670331 29777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:45.671517 29777 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:51:45.674347 29777 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:51:45.674692 29777 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:45.675055 29777 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:51:45.675168 29777 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:45.692572 29839 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e50045e01d1f47b9819454f38124309a)
I20260620 01:51:45.692901 29839 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } }
I20260620 01:51:45.695778 29839 leader_election.cc:290] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e50045e01d1f47b9819454f38124309a (127.20.1.190:44385), 324f486c9a9043f2b98e035d3d5528d2 (127.20.1.189:33769)
I20260620 01:51:45.698719 29124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "324f486c9a9043f2b98e035d3d5528d2" is_pre_election: true
I20260620 01:51:45.704813 29728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e50045e01d1f47b9819454f38124309a" is_pre_election: true
I20260620 01:51:45.705341 29728 raft_consensus.cc:2393] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7acb459ab1bb42aab827fd0d715bc4a2 in current term 3: Already voted for candidate 324f486c9a9043f2b98e035d3d5528d2 in this term.
I20260620 01:51:45.706358 29183 leader_election.cc:304] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7acb459ab1bb42aab827fd0d715bc4a2; no voters: 324f486c9a9043f2b98e035d3d5528d2, e50045e01d1f47b9819454f38124309a
I20260620 01:51:45.706883 29839 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:51:45.711189 29839 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 01:51:45.730832 29777 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:46781
I20260620 01:51:45.731093 29912 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:46781 every 8 connection(s)
I20260620 01:51:45.732990 29777 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:51:45.735721 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 29777
I20260620 01:51:45.785244 29914 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44385
I20260620 01:51:45.785794 29914 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:45.788942 29914 heartbeater.cc:507] Master 127.20.1.190:44385 requested a full tablet report, sending...
I20260620 01:51:45.793735 29913 heartbeater.cc:344] Connected to a master server at 127.20.1.188:43989
I20260620 01:51:45.794435 29913 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:45.796249 29913 heartbeater.cc:507] Master 127.20.1.188:43989 requested a full tablet report, sending...
I20260620 01:51:45.799508 29198 ts_manager.cc:194] Re-registered known tserver with Master: 71bfcd49241b4427bf4049abf8a9b8f6 (127.20.1.129:46781)
I20260620 01:51:45.799751 29718 ts_manager.cc:194] Registered new tserver with Master: 71bfcd49241b4427bf4049abf8a9b8f6 (127.20.1.129:46781)
I20260620 01:51:45.802086 29915 heartbeater.cc:344] Connected to a master server at 127.20.1.189:33769
I20260620 01:51:45.802326 29915 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:45.804664 29915 heartbeater.cc:507] Master 127.20.1.189:33769 requested a full tablet report, sending...
I20260620 01:51:45.808174 29114 ts_manager.cc:194] Re-registered known tserver with Master: 71bfcd49241b4427bf4049abf8a9b8f6 (127.20.1.129:46781)
I20260620 01:51:45.813633 29114 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:47721
I20260620 01:51:45.961743 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29777
I20260620 01:51:45.978816 29908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:46.037600 29728 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [term 3 FOLLOWER]: Refusing update from remote peer 324f486c9a9043f2b98e035d3d5528d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:51:46.038964 29786 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:51:46.046474 29926 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 324f486c9a9043f2b98e035d3d5528d2. Latest consensus state: current_term: 3 leader_uuid: "324f486c9a9043f2b98e035d3d5528d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:46.047039 29926 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:46.065142 29926 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "324f486c9a9043f2b98e035d3d5528d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:46.065522 29926 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50045e01d1f47b9819454f38124309a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:46.069169 29208 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [term 3 FOLLOWER]: Refusing update from remote peer 324f486c9a9043f2b98e035d3d5528d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:51:46.070416 29786 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:51:46.077466 29839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 324f486c9a9043f2b98e035d3d5528d2. Latest consensus state: current_term: 3 leader_uuid: "324f486c9a9043f2b98e035d3d5528d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:46.078027 29839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:46.078970 29782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "324f486c9a9043f2b98e035d3d5528d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:46.079378 29782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: This master's current role is: LEADER
I20260620 01:51:46.077993 29933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "324f486c9a9043f2b98e035d3d5528d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:46.080024 29933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7acb459ab1bb42aab827fd0d715bc4a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:51:46.175999 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 29777
I20260620 01:51:46.203858 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:46781
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--webserver_port=43245
--webserver_interface=127.20.1.129
--time_source=system_unsync
--fault_before_start=1.0 with env {}
I20260620 01:51:46.503782 29930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "324f486c9a9043f2b98e035d3d5528d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50045e01d1f47b9819454f38124309a" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44385 } } peers { permanent_uuid: "324f486c9a9043f2b98e035d3d5528d2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33769 } } peers { permanent_uuid: "7acb459ab1bb42aab827fd0d715bc4a2" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43989 } } }
I20260620 01:51:46.504112 29930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324f486c9a9043f2b98e035d3d5528d2 [sys.catalog]: This master's current role is: LEADER
W20260620 01:51:46.666733 29934 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:46.667229 29934 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:46.667397 29934 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:51:46.667477 29934 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:46.677294 29934 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:46.677575 29934 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:51:46.689102 29934 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:46781
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=43245
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:46.691296 29934 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:46.693972 29934 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:46.708303 29940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:46.708339 29942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:46.708384 29939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:46.710327 29934 server_base.cc:1061] running on GCE node
I20260620 01:51:46.711334 29934 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:46.711531 29934 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:46.711660 29934 hybrid_clock.cc:648] HybridClock initialized: now 1781920306711605 us; error 0 us; skew 500 ppm
I20260620 01:51:46.711990 29934 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:46.715910 29934 webserver.cc:533] Webserver started at http://127.20.1.129:43245/ using document root <none> and password file <none>
I20260620 01:51:46.717007 29934 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:46.717173 29934 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:46.725332 29934 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 01:51:46.729674 29947 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:46.732249 29934 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:51:46.732486 29934 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "71bfcd49241b4427bf4049abf8a9b8f6"
format_stamp: "Formatted at 2026-06-20 01:51:21 on dist-test-slave-wg14"
I20260620 01:51:46.733421 29934 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:46.775934 29934 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:46.777103 29934 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:46.777431 29934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:46.778834 29934 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:51:46.781339 29934 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:51:46.781520 29934 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:46.781670 29934 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:51:46.781847 29934 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
W20260620 01:51:46.782404 29934 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:51:46.782539 29934 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:51:46.782640 29934 fault_injection.cc:59] injecting fault for kudu (pid 29934): FLAGS_fault_before_start (process will exit)
I20260620 01:51:46.800725 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:46781
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--webserver_port=43245
--webserver_interface=127.20.1.129
--time_source=system_unsync with env {}
I20260620 01:51:47.033535 29676 heartbeater.cc:499] Master 127.20.1.189:33769 was elected leader, sending a full tablet report...
W20260620 01:51:47.233783 29955 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:51:47.234222 29955 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:51:47.234357 29955 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:51:47.244056 29955 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:51:47.244318 29955 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:51:47.256852 29955 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:46781
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=43245
--tserver_master_addrs=127.20.1.190:44385,127.20.1.189:33769,127.20.1.188:43989
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:51:47.258880 29955 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:51:47.261176 29955 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:51:47.276789 29961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:47.277034 29963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:51:47.277011 29960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:51:47.278090 29955 server_base.cc:1061] running on GCE node
I20260620 01:51:47.279089 29955 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:51:47.279353 29955 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:51:47.279642 29955 hybrid_clock.cc:648] HybridClock initialized: now 1781920307279429 us; error 0 us; skew 500 ppm
I20260620 01:51:47.280229 29955 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:51:47.284281 29955 webserver.cc:533] Webserver started at http://127.20.1.129:43245/ using document root <none> and password file <none>
I20260620 01:51:47.285485 29955 fs_manager.cc:362] Metadata directory not provided
I20260620 01:51:47.285626 29955 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:51:47.293692 29955 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:51:47.298437 29968 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:47.300865 29955 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:51:47.301123 29955 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "71bfcd49241b4427bf4049abf8a9b8f6"
format_stamp: "Formatted at 2026-06-20 01:51:21 on dist-test-slave-wg14"
I20260620 01:51:47.302048 29955 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:51:47.322618 29955 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:51:47.323900 29955 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:51:47.324334 29955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:51:47.325734 29955 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:51:47.328233 29955 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:51:47.328361 29955 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:47.328773 29955 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:51:47.328869 29955 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:51:47.379525 29955 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:46781
I20260620 01:51:47.379680 30080 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:46781 every 8 connection(s)
I20260620 01:51:47.382120 29955 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:51:47.389508 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 29955
I20260620 01:51:47.389966 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29955
I20260620 01:51:47.406165 29540 heartbeater.cc:499] Master 127.20.1.189:33769 was elected leader, sending a full tablet report...
I20260620 01:51:47.412452 30076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:47.518258 30081 heartbeater.cc:344] Connected to a master server at 127.20.1.188:43989
I20260620 01:51:47.519059 30081 heartbeater.cc:461] Registering TS with master...
I20260620 01:51:47.520449 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 29955
I20260620 01:51:47.536355 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29406
W20260620 01:51:47.537274 30092 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:51:47.396463 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:38581 (local address 127.20.1.189:33769)
0620 01:51:47.399734 (+ 3271us) server_negotiation.cc:207] Beginning negotiation
0620 01:51:47.399741 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 01:51:47.411330 (+ 11589us) server_negotiation.cc:408] Connection header received
0620 01:51:47.411417 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:51:47.411421 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:51:47.411516 (+ 95us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:51:47.411655 (+ 139us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:51:47.413052 (+ 1397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:51:47.415411 (+ 2359us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:51:47.536173 (+120762us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:38581: BlockingRecv error: recv got EOF from 127.20.1.129:38581 (error 108)
Metrics: {"server-negotiator.queue_time_us":3099,"thread_start_us":2944,"threads_started":1}
I20260620 01:51:47.546859 29533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:47.647341 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 29406
I20260620 01:51:47.664628 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29547
I20260620 01:51:47.678170 29669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:47.780524 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 29547
I20260620 01:51:47.795372 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29686
I20260620 01:51:47.804047 29748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:47.930204 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 29686
I20260620 01:51:47.952654 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29068
I20260620 01:51:47.954584 29144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:48.084142 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 29068
W20260620 01:51:48.103947 29185 connection.cc:573] server connection from 127.0.0.1:47208 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:51:48.104508 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29151
I20260620 01:51:48.105928 29228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:51:48.234926 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 29151
Shutting down hive metastore.
[ OK ] ExternalMiniClusterTest.TestBasicOperation/3 (53806 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/4
2026-06-20T01:51:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:51:48Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/security.properties
2026-06-20 01:51:58: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260620 01:52:11.758873 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40603
--webserver_interface=127.20.1.190
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:36667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463 with env {}
W20260620 01:52:12.212634 30393 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:12.213109 30393 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:12.213191 30393 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:12.224113 30393 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:52:12.224287 30393 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:12.224354 30393 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:52:12.224399 30393 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:52:12.236675 30393 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:12.238611 30393 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:12.240990 30393 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:12.253377 30399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:12.253403 30398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:12.254453 30401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:12.255630 30393 server_base.cc:1061] running on GCE node
I20260620 01:52:12.256677 30393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:12.258844 30393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:12.260476 30393 hybrid_clock.cc:648] HybridClock initialized: now 1781920332260354 us; error 129 us; skew 500 ppm
I20260620 01:52:12.261171 30393 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:12.264791 30393 webserver.cc:533] Webserver started at http://127.20.1.190:43885/ using document root <none> and password file <none>
I20260620 01:52:12.265878 30393 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:12.266026 30393 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:12.266445 30393 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:52:12.269577 30393 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "a15143148e7b4afa83dc0ae5772ad3b3"
format_stamp: "Formatted at 2026-06-20 01:52:12 on dist-test-slave-wg14"
I20260620 01:52:12.270618 30393 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "a15143148e7b4afa83dc0ae5772ad3b3"
format_stamp: "Formatted at 2026-06-20 01:52:12 on dist-test-slave-wg14"
I20260620 01:52:12.279511 30393 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.011s sys 0.001s
I20260620 01:52:12.285660 30407 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:12.287993 30393 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:52:12.288275 30393 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "a15143148e7b4afa83dc0ae5772ad3b3"
format_stamp: "Formatted at 2026-06-20 01:52:12 on dist-test-slave-wg14"
I20260620 01:52:12.288597 30393 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:12.310004 30393 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:12.311411 30393 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:12.311849 30393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:12.340951 30393 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:40603
I20260620 01:52:12.341058 30458 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:40603 every 8 connection(s)
I20260620 01:52:12.342976 30393 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:52:12.349645 30459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:52:12.350152 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 30393
I20260620 01:52:12.350395 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:52:12.351974 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:39115
--webserver_interface=127.20.1.189
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:36667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463 with env {}
I20260620 01:52:12.361260 30459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:52:12.389955 30459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:52:12.392009 30408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:39115: connect: Connection refused (error 111)
W20260620 01:52:12.394737 30459 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:39115: Network error: Client connection negotiation failed: client connection to 127.20.1.189:39115: connect: Connection refused (error 111)
I20260620 01:52:12.444358 30459 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } attempt: 1
W20260620 01:52:12.447953 30459 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:39115: Network error: Client connection negotiation failed: client connection to 127.20.1.189:39115: connect: Connection refused (error 111)
I20260620 01:52:12.516353 30459 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } attempt: 2
W20260620 01:52:12.518507 30459 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:39115: Network error: Client connection negotiation failed: client connection to 127.20.1.189:39115: connect: Connection refused (error 111)
I20260620 01:52:12.609845 30459 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } attempt: 3
W20260620 01:52:12.612680 30459 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:39115: Network error: Client connection negotiation failed: client connection to 127.20.1.189:39115: connect: Connection refused (error 111)
I20260620 01:52:12.756086 30459 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } attempt: 4
W20260620 01:52:12.758786 30459 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:39115: Network error: Client connection negotiation failed: client connection to 127.20.1.189:39115: connect: Connection refused (error 111)
W20260620 01:52:12.787580 30461 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:12.788192 30461 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:12.788363 30461 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:12.800045 30461 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:52:12.800184 30461 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:12.800246 30461 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:52:12.800297 30461 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:52:12.812294 30461 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:39115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:12.814358 30461 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:12.816978 30461 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:12.829062 30472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:12.829262 30469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:12.829044 30470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:12.830899 30461 server_base.cc:1061] running on GCE node
I20260620 01:52:12.832026 30461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:12.834102 30461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:12.835546 30461 hybrid_clock.cc:648] HybridClock initialized: now 1781920332835341 us; error 86 us; skew 500 ppm
I20260620 01:52:12.836366 30461 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:12.840378 30461 webserver.cc:533] Webserver started at http://127.20.1.189:41555/ using document root <none> and password file <none>
I20260620 01:52:12.841516 30461 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:12.841704 30461 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:12.842377 30461 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:52:12.845649 30461 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "e887a75e4e8047708198e5dcc9bc2b6b"
format_stamp: "Formatted at 2026-06-20 01:52:12 on dist-test-slave-wg14"
I20260620 01:52:12.846698 30461 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "e887a75e4e8047708198e5dcc9bc2b6b"
format_stamp: "Formatted at 2026-06-20 01:52:12 on dist-test-slave-wg14"
I20260620 01:52:12.854715 30461 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260620 01:52:12.860766 30478 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:12.863348 30461 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:52:12.863617 30461 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "e887a75e4e8047708198e5dcc9bc2b6b"
format_stamp: "Formatted at 2026-06-20 01:52:12 on dist-test-slave-wg14"
I20260620 01:52:12.863922 30461 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:12.890375 30461 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:12.891793 30461 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:12.892199 30461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:12.920156 30461 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:39115
I20260620 01:52:12.920156 30529 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:39115 every 8 connection(s)
I20260620 01:52:12.922293 30461 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:52:12.928061 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 30461
I20260620 01:52:12.928443 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:52:12.929808 30530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:52:12.930054 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:46463
--webserver_interface=127.20.1.188
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:36667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463 with env {}
I20260620 01:52:12.938318 30530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:52:12.957816 30530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:52:12.972070 30530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:52:12.975075 30480 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46463: connect: Connection refused (error 111)
W20260620 01:52:12.979243 30530 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46463: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46463: connect: Connection refused (error 111)
I20260620 01:52:13.028784 30530 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } attempt: 1
W20260620 01:52:13.032522 30530 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46463: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46463: connect: Connection refused (error 111)
I20260620 01:52:13.058254 30459 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } attempt: 5
I20260620 01:52:13.068434 30459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:52:13.073685 30459 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46463: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46463: connect: Connection refused (error 111)
I20260620 01:52:13.100893 30530 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } attempt: 2
W20260620 01:52:13.104012 30530 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46463: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46463: connect: Connection refused (error 111)
I20260620 01:52:13.125119 30459 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } attempt: 1
W20260620 01:52:13.128535 30459 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46463: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46463: connect: Connection refused (error 111)
I20260620 01:52:13.195533 30530 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } attempt: 3
I20260620 01:52:13.196875 30459 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } attempt: 2
W20260620 01:52:13.199762 30530 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46463: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46463: connect: Connection refused (error 111)
W20260620 01:52:13.199784 30459 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46463: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46463: connect: Connection refused (error 111)
I20260620 01:52:13.306114 30459 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } attempt: 3
W20260620 01:52:13.308626 30459 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46463: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46463: connect: Connection refused (error 111)
I20260620 01:52:13.343257 30530 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } attempt: 4
W20260620 01:52:13.346379 30530 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46463: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46463: connect: Connection refused (error 111)
W20260620 01:52:13.429375 30532 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:13.430756 30532 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:13.430955 30532 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:13.442605 30532 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:52:13.442755 30532 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:13.442821 30532 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:52:13.442870 30532 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:52:13.456293 30532 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:46463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:13.458201 30532 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:13.460599 30532 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:13.473523 30541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:13.473477 30542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:13.474625 30544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:13.476722 30532 server_base.cc:1061] running on GCE node
I20260620 01:52:13.478457 30532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:13.481339 30532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:13.482774 30532 hybrid_clock.cc:648] HybridClock initialized: now 1781920333482689 us; error 77 us; skew 500 ppm
I20260620 01:52:13.483422 30532 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:13.485929 30459 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } attempt: 4
I20260620 01:52:13.486873 30532 webserver.cc:533] Webserver started at http://127.20.1.188:44447/ using document root <none> and password file <none>
I20260620 01:52:13.488204 30532 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:13.488399 30532 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260620 01:52:13.488391 30459 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46463: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46463: connect: Connection refused (error 111)
I20260620 01:52:13.489027 30532 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:52:13.493845 30532 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "5a3fba60e67d4472940e013e0b864158"
format_stamp: "Formatted at 2026-06-20 01:52:13 on dist-test-slave-wg14"
I20260620 01:52:13.495710 30532 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "5a3fba60e67d4472940e013e0b864158"
format_stamp: "Formatted at 2026-06-20 01:52:13 on dist-test-slave-wg14"
I20260620 01:52:13.504814 30532 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260620 01:52:13.510838 30550 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:13.512902 30532 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260620 01:52:13.513350 30532 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "5a3fba60e67d4472940e013e0b864158"
format_stamp: "Formatted at 2026-06-20 01:52:13 on dist-test-slave-wg14"
I20260620 01:52:13.513659 30532 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:13.558687 30532 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:13.559877 30532 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:13.560220 30532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:13.588377 30532 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:46463
I20260620 01:52:13.588392 30601 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:46463 every 8 connection(s)
I20260620 01:52:13.590418 30532 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:52:13.597409 30602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:52:13.597738 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 30532
I20260620 01:52:13.597977 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:52:13.609689 30602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:52:13.635722 30602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:52:13.645754 30530 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } attempt: 5
I20260620 01:52:13.646862 30602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:52:13.663565 30602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158: Bootstrap starting.
I20260620 01:52:13.664048 30530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b: Bootstrap starting.
I20260620 01:52:13.668666 30602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158: Neither blocks nor log segments found. Creating new log.
I20260620 01:52:13.668664 30530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b: Neither blocks nor log segments found. Creating new log.
I20260620 01:52:13.670164 30602 log.cc:826] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158: Log is configured to *not* fsync() on all Append() calls
I20260620 01:52:13.670192 30530 log.cc:826] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b: Log is configured to *not* fsync() on all Append() calls
I20260620 01:52:13.677213 30602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158: No bootstrap required, opened a new log
I20260620 01:52:13.681726 30530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b: No bootstrap required, opened a new log
I20260620 01:52:13.686882 30602 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:13.687422 30602 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:52:13.687553 30602 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3fba60e67d4472940e013e0b864158, State: Initialized, Role: FOLLOWER
I20260620 01:52:13.688582 30602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:13.690236 30612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:13.690728 30612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:13.691442 30530 raft_consensus.cc:359] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:13.691874 30530 raft_consensus.cc:385] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:52:13.692037 30530 raft_consensus.cc:740] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e887a75e4e8047708198e5dcc9bc2b6b, State: Initialized, Role: FOLLOWER
I20260620 01:52:13.692137 30602 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:52:13.693017 30530 consensus_queue.cc:260] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:13.697067 30613 sys_catalog.cc:455] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:13.697696 30613 sys_catalog.cc:458] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:13.698403 30530 sys_catalog.cc:565] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:52:13.711907 30461 master_runner.cc:187] Error getting master registration for 127.20.1.190:40603: OK, instance_id { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" instance_seqno: 1781920332334159 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:52:13.721696 30461 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:52:13.712672 30532 master_runner.cc:187] Error getting master registration for 127.20.1.190:40603: OK, instance_id { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" instance_seqno: 1781920332334159 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:52:13.725131 30532 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:52:13.728473 30635 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:52:13.728708 30635 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 01:52:13.729288 30633 catalog_manager.cc:1593] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:52:13.729518 30633 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:52:13.765729 30459 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } attempt: 5
I20260620 01:52:13.785960 30459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: Bootstrap starting.
I20260620 01:52:13.792309 30459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: Neither blocks nor log segments found. Creating new log.
I20260620 01:52:13.794914 30459 log.cc:826] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: Log is configured to *not* fsync() on all Append() calls
I20260620 01:52:13.804248 30459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: No bootstrap required, opened a new log
W20260620 01:52:13.813433 30556 tablet.cc:2406] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:52:13.814648 30459 raft_consensus.cc:359] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:13.815218 30459 raft_consensus.cc:385] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:52:13.815357 30459 raft_consensus.cc:740] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15143148e7b4afa83dc0ae5772ad3b3, State: Initialized, Role: FOLLOWER
I20260620 01:52:13.816394 30459 consensus_queue.cc:260] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:13.820673 30639 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:13.821256 30639 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:13.822631 30459 sys_catalog.cc:565] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:52:13.845575 30650 catalog_manager.cc:1593] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 01:52:13.850495 30393 master_runner.cc:264] No leader master found from master a15143148e7b4afa83dc0ae5772ad3b3
I20260620 01:52:13.852560 30393 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:52:13.853434 30650 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 01:52:13.896212 30484 tablet.cc:2406] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:52:13.898640 30613 raft_consensus.cc:493] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:52:13.899238 30613 raft_consensus.cc:515] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:13.902318 30613 leader_election.cc:290] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a15143148e7b4afa83dc0ae5772ad3b3 (127.20.1.190:40603), 5a3fba60e67d4472940e013e0b864158 (127.20.1.188:46463)
I20260620 01:52:13.903163 30434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" is_pre_election: true
I20260620 01:52:13.904135 30434 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e887a75e4e8047708198e5dcc9bc2b6b in term 0.
I20260620 01:52:13.905179 30481 leader_election.cc:304] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15143148e7b4afa83dc0ae5772ad3b3, e887a75e4e8047708198e5dcc9bc2b6b; no voters:
I20260620 01:52:13.906963 30577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3fba60e67d4472940e013e0b864158" is_pre_election: true
I20260620 01:52:13.907703 30577 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e887a75e4e8047708198e5dcc9bc2b6b in term 0.
I20260620 01:52:13.909153 30613 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:52:13.909423 30613 raft_consensus.cc:493] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:52:13.909547 30613 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:52:13.913489 30613 raft_consensus.cc:515] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:13.915243 30434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15143148e7b4afa83dc0ae5772ad3b3"
I20260620 01:52:13.915549 30434 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:52:13.919860 30434 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e887a75e4e8047708198e5dcc9bc2b6b in term 1.
I20260620 01:52:13.924208 30481 leader_election.cc:304] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15143148e7b4afa83dc0ae5772ad3b3, e887a75e4e8047708198e5dcc9bc2b6b; no voters:
I20260620 01:52:13.929953 30577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3fba60e67d4472940e013e0b864158"
I20260620 01:52:13.930270 30577 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:52:13.934893 30577 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e887a75e4e8047708198e5dcc9bc2b6b in term 1.
I20260620 01:52:13.936534 30613 leader_election.cc:290] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [CANDIDATE]: Term 1 election: Requested vote from peers a15143148e7b4afa83dc0ae5772ad3b3 (127.20.1.190:40603), 5a3fba60e67d4472940e013e0b864158 (127.20.1.188:46463)
I20260620 01:52:13.937007 30613 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:52:13.937191 30613 raft_consensus.cc:697] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 1 LEADER]: Becoming Leader. State: Replica: e887a75e4e8047708198e5dcc9bc2b6b, State: Running, Role: LEADER
I20260620 01:52:13.938170 30613 consensus_queue.cc:237] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:13.973342 30652 sys_catalog.cc:455] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e887a75e4e8047708198e5dcc9bc2b6b. Latest consensus state: current_term: 1 leader_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:13.973917 30652 sys_catalog.cc:458] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: This master's current role is: LEADER
I20260620 01:52:13.988899 30654 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:52:13.992707 30654 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:52:14.024819 30577 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 1 FOLLOWER]: Refusing update from remote peer e887a75e4e8047708198e5dcc9bc2b6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:52:14.032814 30652 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:52:14.045608 30434 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 1 FOLLOWER]: Refusing update from remote peer e887a75e4e8047708198e5dcc9bc2b6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:52:14.047465 30652 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260620 01:52:14.067786 30413 tablet.cc:2406] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:52:14.085672 30639 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e887a75e4e8047708198e5dcc9bc2b6b. Latest consensus state: current_term: 1 leader_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:14.086081 30639 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:14.092893 30655 mvcc.cc:204] Tried to move back new op lower bound from 7298745688135897088 to 7298745687835807744. Current Snapshot: MvccSnapshot[applied={T|T < 7298745688135897088}]
I20260620 01:52:14.098253 30612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e887a75e4e8047708198e5dcc9bc2b6b. Latest consensus state: current_term: 1 leader_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:14.098680 30612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:14.110700 30612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:14.111032 30612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:14.111629 30639 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:14.111961 30639 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:14.123117 30654 catalog_manager.cc:1382] Generated new cluster ID: 368ec3da3f0d467c970a416c3fb7fa92
I20260620 01:52:14.123353 30654 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:52:14.139104 30663 sys_catalog.cc:455] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:14.139870 30663 sys_catalog.cc:458] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: This master's current role is: LEADER
I20260620 01:52:14.140776 30613 sys_catalog.cc:455] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:14.141371 30613 sys_catalog.cc:458] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: This master's current role is: LEADER
I20260620 01:52:14.162669 30654 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:52:14.168418 30654 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:52:14.207892 30654 catalog_manager.cc:6069] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b: Generated new TSK 0
I20260620 01:52:14.209751 30654 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:52:14.236550 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--builtin_ntp_servers=127.20.1.148:36667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:52:14.731632 30635 catalog_manager.cc:1294] Loaded cluster ID: 368ec3da3f0d467c970a416c3fb7fa92
I20260620 01:52:14.731812 30635 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158: loading cluster ID for follower catalog manager: success
I20260620 01:52:14.735754 30635 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158: acquiring CA information for follower catalog manager: success
I20260620 01:52:14.738446 30635 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:52:14.855975 30650 catalog_manager.cc:1294] Loaded cluster ID: 368ec3da3f0d467c970a416c3fb7fa92
I20260620 01:52:14.856109 30650 catalog_manager.cc:1587] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: loading cluster ID for follower catalog manager: success
I20260620 01:52:14.859760 30650 catalog_manager.cc:1609] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: acquiring CA information for follower catalog manager: success
W20260620 01:52:14.860656 30664 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:14.861083 30664 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:14.861927 30664 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260620 01:52:14.862723 30650 catalog_manager.cc:1637] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 01:52:14.872925 30664 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:14.873191 30664 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:52:14.885780 30664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:14.889158 30664 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:14.892086 30664 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:14.906101 30669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:14.906353 30670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:14.906131 30672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:14.907661 30664 server_base.cc:1061] running on GCE node
I20260620 01:52:14.908584 30664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:14.909926 30664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:14.911276 30664 hybrid_clock.cc:648] HybridClock initialized: now 1781920334911218 us; error 75 us; skew 500 ppm
I20260620 01:52:14.911774 30664 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:14.914971 30664 webserver.cc:533] Webserver started at http://127.20.1.129:43565/ using document root <none> and password file <none>
I20260620 01:52:14.916006 30664 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:14.916179 30664 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:14.916611 30664 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:52:14.922484 30664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "21330062fe2642cba2c0c86f7b17d5ed"
format_stamp: "Formatted at 2026-06-20 01:52:14 on dist-test-slave-wg14"
I20260620 01:52:14.923969 30664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "21330062fe2642cba2c0c86f7b17d5ed"
format_stamp: "Formatted at 2026-06-20 01:52:14 on dist-test-slave-wg14"
I20260620 01:52:14.933020 30664 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260620 01:52:14.938560 30678 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:14.940930 30664 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:52:14.941193 30664 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "21330062fe2642cba2c0c86f7b17d5ed"
format_stamp: "Formatted at 2026-06-20 01:52:14 on dist-test-slave-wg14"
I20260620 01:52:14.941500 30664 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:14.962105 30664 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:14.963570 30664 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:14.964211 30664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:14.966293 30664 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:52:14.970010 30664 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:52:14.970197 30664 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:14.970324 30664 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:52:14.970396 30664 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:15.017827 30664 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:33629
I20260620 01:52:15.017895 30790 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:33629 every 8 connection(s)
I20260620 01:52:15.019950 30664 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:52:15.022040 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 30664
I20260620 01:52:15.022286 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:52:15.034416 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--builtin_ntp_servers=127.20.1.148:36667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:52:15.059854 30793 heartbeater.cc:344] Connected to a master server at 127.20.1.189:39115
I20260620 01:52:15.059990 30792 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40603
I20260620 01:52:15.060693 30793 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:15.060691 30792 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:15.063215 30793 heartbeater.cc:507] Master 127.20.1.189:39115 requested a full tablet report, sending...
I20260620 01:52:15.063414 30792 heartbeater.cc:507] Master 127.20.1.190:40603 requested a full tablet report, sending...
I20260620 01:52:15.067368 30495 ts_manager.cc:194] Registered new tserver with Master: 21330062fe2642cba2c0c86f7b17d5ed (127.20.1.129:33629)
I20260620 01:52:15.067384 30424 ts_manager.cc:194] Registered new tserver with Master: 21330062fe2642cba2c0c86f7b17d5ed (127.20.1.129:33629)
I20260620 01:52:15.068373 30791 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46463
I20260620 01:52:15.068617 30791 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:15.069703 30495 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:52269
I20260620 01:52:15.070006 30791 heartbeater.cc:507] Master 127.20.1.188:46463 requested a full tablet report, sending...
I20260620 01:52:15.073386 30567 ts_manager.cc:194] Registered new tserver with Master: 21330062fe2642cba2c0c86f7b17d5ed (127.20.1.129:33629)
W20260620 01:52:15.551483 30801 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:15.551882 30801 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:15.552002 30801 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:15.563429 30801 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:15.563635 30801 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:52:15.580406 30801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:15.583617 30801 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:15.588079 30801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:15.605561 30808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:15.606055 30807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:15.609727 30810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:15.610044 30801 server_base.cc:1061] running on GCE node
I20260620 01:52:15.611937 30801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:15.613364 30801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:15.615275 30801 hybrid_clock.cc:648] HybridClock initialized: now 1781920335615138 us; error 62 us; skew 500 ppm
I20260620 01:52:15.617425 30801 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:15.622996 30801 webserver.cc:533] Webserver started at http://127.20.1.130:43073/ using document root <none> and password file <none>
I20260620 01:52:15.624228 30801 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:15.624419 30801 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:15.625000 30801 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:52:15.629242 30801 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/data/instance:
uuid: "7f5ceedc91a6418497b6a16dff4ed5d8"
format_stamp: "Formatted at 2026-06-20 01:52:15 on dist-test-slave-wg14"
I20260620 01:52:15.633376 30801 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance:
uuid: "7f5ceedc91a6418497b6a16dff4ed5d8"
format_stamp: "Formatted at 2026-06-20 01:52:15 on dist-test-slave-wg14"
I20260620 01:52:15.646978 30801 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.009s sys 0.004s
I20260620 01:52:15.655355 30816 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:15.658360 30801 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.004s sys 0.003s
I20260620 01:52:15.658680 30801 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "7f5ceedc91a6418497b6a16dff4ed5d8"
format_stamp: "Formatted at 2026-06-20 01:52:15 on dist-test-slave-wg14"
I20260620 01:52:15.659044 30801 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:15.683732 30801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:15.684836 30801 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:15.685164 30801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:15.686419 30801 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:52:15.688882 30801 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:52:15.689038 30801 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:15.689138 30801 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:52:15.689208 30801 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:15.737334 30801 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:37611
I20260620 01:52:15.737435 30928 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:37611 every 8 connection(s)
I20260620 01:52:15.739053 30801 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:52:15.745779 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 30801
I20260620 01:52:15.746037 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
I20260620 01:52:15.767697 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--builtin_ntp_servers=127.20.1.148:36667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:52:15.779994 30931 heartbeater.cc:344] Connected to a master server at 127.20.1.189:39115
I20260620 01:52:15.780565 30931 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:15.782088 30931 heartbeater.cc:507] Master 127.20.1.189:39115 requested a full tablet report, sending...
I20260620 01:52:15.785168 30495 ts_manager.cc:194] Registered new tserver with Master: 7f5ceedc91a6418497b6a16dff4ed5d8 (127.20.1.130:37611)
I20260620 01:52:15.786509 30495 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.130:41623
I20260620 01:52:15.788962 30930 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40603
I20260620 01:52:15.789547 30930 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:15.790827 30929 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46463
I20260620 01:52:15.791234 30929 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:15.792963 30929 heartbeater.cc:507] Master 127.20.1.188:46463 requested a full tablet report, sending...
I20260620 01:52:15.792981 30930 heartbeater.cc:507] Master 127.20.1.190:40603 requested a full tablet report, sending...
I20260620 01:52:15.794574 30567 ts_manager.cc:194] Registered new tserver with Master: 7f5ceedc91a6418497b6a16dff4ed5d8 (127.20.1.130:37611)
I20260620 01:52:15.795322 30424 ts_manager.cc:194] Registered new tserver with Master: 7f5ceedc91a6418497b6a16dff4ed5d8 (127.20.1.130:37611)
I20260620 01:52:16.074694 30793 heartbeater.cc:499] Master 127.20.1.189:39115 was elected leader, sending a full tablet report...
W20260620 01:52:16.291617 30939 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:16.292047 30939 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:16.292174 30939 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:16.302557 30939 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:16.302917 30939 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:52:16.315105 30939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:16.317077 30939 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:16.319618 30939 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:16.334084 30944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:16.333894 30945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:16.334904 30947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:16.336647 30939 server_base.cc:1061] running on GCE node
I20260620 01:52:16.337718 30939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:16.339473 30939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:16.340850 30939 hybrid_clock.cc:648] HybridClock initialized: now 1781920336340784 us; error 56 us; skew 500 ppm
I20260620 01:52:16.341373 30939 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:16.345661 30939 webserver.cc:533] Webserver started at http://127.20.1.131:42635/ using document root <none> and password file <none>
I20260620 01:52:16.346982 30939 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:16.347209 30939 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:16.347745 30939 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:52:16.353186 30939 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/data/instance:
uuid: "e1f9b3cf197948de83c943800b4c90ad"
format_stamp: "Formatted at 2026-06-20 01:52:16 on dist-test-slave-wg14"
I20260620 01:52:16.356271 30939 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance:
uuid: "e1f9b3cf197948de83c943800b4c90ad"
format_stamp: "Formatted at 2026-06-20 01:52:16 on dist-test-slave-wg14"
I20260620 01:52:16.364367 30939 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260620 01:52:16.370232 30953 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:16.373116 30939 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:52:16.373554 30939 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "e1f9b3cf197948de83c943800b4c90ad"
format_stamp: "Formatted at 2026-06-20 01:52:16 on dist-test-slave-wg14"
I20260620 01:52:16.374006 30939 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:16.393819 30939 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:16.395594 30939 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:16.395937 30939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:16.397450 30939 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:52:16.399801 30939 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:52:16.399994 30939 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:16.400130 30939 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:52:16.400197 30939 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:16.446374 30939 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:45427
I20260620 01:52:16.446424 31065 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:45427 every 8 connection(s)
I20260620 01:52:16.448309 30939 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:52:16.453325 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 30939
I20260620 01:52:16.453558 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
I20260620 01:52:16.480309 31068 heartbeater.cc:344] Connected to a master server at 127.20.1.189:39115
I20260620 01:52:16.480880 31068 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:16.482394 31068 heartbeater.cc:507] Master 127.20.1.189:39115 requested a full tablet report, sending...
I20260620 01:52:16.483382 31066 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46463
I20260620 01:52:16.483546 31066 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:16.484571 31066 heartbeater.cc:507] Master 127.20.1.188:46463 requested a full tablet report, sending...
I20260620 01:52:16.485334 31067 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40603
I20260620 01:52:16.485351 30495 ts_manager.cc:194] Registered new tserver with Master: e1f9b3cf197948de83c943800b4c90ad (127.20.1.131:45427)
I20260620 01:52:16.485563 31067 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:16.486075 30567 ts_manager.cc:194] Registered new tserver with Master: e1f9b3cf197948de83c943800b4c90ad (127.20.1.131:45427)
I20260620 01:52:16.486200 31067 heartbeater.cc:507] Master 127.20.1.190:40603 requested a full tablet report, sending...
I20260620 01:52:16.486469 30495 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.131:38743
I20260620 01:52:16.487886 30424 ts_manager.cc:194] Registered new tserver with Master: e1f9b3cf197948de83c943800b4c90ad (127.20.1.131:45427)
I20260620 01:52:16.495954 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:52:16.511070 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:52:16.511353 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 30393
I20260620 01:52:16.512634 30454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:16.643366 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 30393
I20260620 01:52:16.663058 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40603
--webserver_interface=127.20.1.190
--webserver_port=43885
--builtin_ntp_servers=127.20.1.148:36667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463 with env {}
W20260620 01:52:16.742636 30481 consensus_peers.cc:597] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b -> Peer a15143148e7b4afa83dc0ae5772ad3b3 (127.20.1.190:40603): Couldn't send request to peer a15143148e7b4afa83dc0ae5772ad3b3. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:40603: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 01:52:16.795987 30931 heartbeater.cc:499] Master 127.20.1.189:39115 was elected leader, sending a full tablet report...
W20260620 01:52:16.799261 30930 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:40603 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:40603: connect: Connection refused (error 111)
W20260620 01:52:17.079531 30792 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:40603 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:40603: connect: Connection refused (error 111)
W20260620 01:52:17.141007 31077 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:17.141589 31077 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:17.141746 31077 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:17.158093 31077 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:52:17.158313 31077 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:17.158401 31077 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:52:17.158464 31077 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:52:17.180869 31077 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=43885
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:17.185124 31077 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:17.189101 31077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:17.203718 31087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:17.213104 31086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:17.215265 31077 server_base.cc:1061] running on GCE node
W20260620 01:52:17.215441 31089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:17.216967 31077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:17.219244 31077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:17.222997 31077 hybrid_clock.cc:648] HybridClock initialized: now 1781920337222768 us; error 200 us; skew 500 ppm
I20260620 01:52:17.223584 31077 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:17.227072 31077 webserver.cc:533] Webserver started at http://127.20.1.190:43885/ using document root <none> and password file <none>
I20260620 01:52:17.229805 31077 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:17.230072 31077 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:17.239663 31077 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.002s sys 0.005s
I20260620 01:52:17.245879 31095 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:17.249872 31077 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.005s sys 0.001s
I20260620 01:52:17.250155 31077 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "a15143148e7b4afa83dc0ae5772ad3b3"
format_stamp: "Formatted at 2026-06-20 01:52:12 on dist-test-slave-wg14"
I20260620 01:52:17.252082 31077 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:17.294364 31096 sys_catalog.cc:455] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:17.294703 31096 sys_catalog.cc:458] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: This master's current role is: LEADER
I20260620 01:52:17.294682 31077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:17.296495 31077 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:17.297163 31077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:17.348049 31077 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:40603
I20260620 01:52:17.348145 31152 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:40603 every 8 connection(s)
I20260620 01:52:17.350277 31077 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:52:17.361060 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 31077
I20260620 01:52:17.361791 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 30664
I20260620 01:52:17.368963 31153 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:52:17.380789 31153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: Bootstrap starting.
I20260620 01:52:17.397703 30786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:17.411831 31153 log.cc:826] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: Log is configured to *not* fsync() on all Append() calls
I20260620 01:52:17.426030 31153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:52:17.426723 31153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: Bootstrap complete.
I20260620 01:52:17.439397 31153 raft_consensus.cc:359] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:17.440793 31153 raft_consensus.cc:740] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15143148e7b4afa83dc0ae5772ad3b3, State: Initialized, Role: FOLLOWER
I20260620 01:52:17.441911 31153 consensus_queue.cc:260] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:17.443868 31159 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:17.444367 31159 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:17.445379 31153 sys_catalog.cc:565] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:52:17.488672 31170 catalog_manager.cc:1294] Loaded cluster ID: 368ec3da3f0d467c970a416c3fb7fa92
I20260620 01:52:17.489696 31170 catalog_manager.cc:1587] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: loading cluster ID for follower catalog manager: success
I20260620 01:52:17.490900 31068 heartbeater.cc:499] Master 127.20.1.189:39115 was elected leader, sending a full tablet report...
I20260620 01:52:17.496528 31170 catalog_manager.cc:1609] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: acquiring CA information for follower catalog manager: success
I20260620 01:52:17.511873 31170 catalog_manager.cc:1637] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:52:17.528667 31118 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e1f9b3cf197948de83c943800b4c90ad" instance_seqno: 1781920336433648) as {username='slave'} at 127.20.1.131:33011; Asking this server to re-register.
I20260620 01:52:17.530118 31067 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:17.533576 31118 ts_manager.cc:194] Registered new tserver with Master: e1f9b3cf197948de83c943800b4c90ad (127.20.1.131:45427)
I20260620 01:52:17.576715 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 30664
I20260620 01:52:17.598927 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:33629
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--webserver_port=43565
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:36667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:52:17.785816 31159 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e887a75e4e8047708198e5dcc9bc2b6b. Latest consensus state: current_term: 1 leader_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:17.786235 31159 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:17.819392 30930 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40603
I20260620 01:52:17.821610 31118 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7f5ceedc91a6418497b6a16dff4ed5d8" instance_seqno: 1781920335725521) as {username='slave'} at 127.20.1.130:45005; Asking this server to re-register.
I20260620 01:52:17.822541 30930 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:17.824978 31118 ts_manager.cc:194] Registered new tserver with Master: 7f5ceedc91a6418497b6a16dff4ed5d8 (127.20.1.130:37611)
W20260620 01:52:18.087069 31178 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:18.087713 31178 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:18.087908 31178 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:18.098388 31178 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:18.098664 31178 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:52:18.112498 31178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:33629
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=43565
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:18.114858 31178 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:18.117527 31178 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:18.133021 31185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:18.132917 31184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:18.132946 31187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:18.133978 31178 server_base.cc:1061] running on GCE node
I20260620 01:52:18.134832 31178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:18.136248 31178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:18.137683 31178 hybrid_clock.cc:648] HybridClock initialized: now 1781920338137595 us; error 51 us; skew 500 ppm
I20260620 01:52:18.138197 31178 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:18.141474 31178 webserver.cc:533] Webserver started at http://127.20.1.129:43565/ using document root <none> and password file <none>
I20260620 01:52:18.142689 31178 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:18.142892 31178 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:18.151247 31178 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:52:18.155427 31193 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:18.157811 31178 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:52:18.158061 31178 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "21330062fe2642cba2c0c86f7b17d5ed"
format_stamp: "Formatted at 2026-06-20 01:52:14 on dist-test-slave-wg14"
I20260620 01:52:18.158972 31178 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:18.188299 31178 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:18.189810 31178 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:18.190244 31178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:18.191614 31178 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:52:18.194593 31178 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:52:18.194739 31178 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:18.194893 31178 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:52:18.194957 31178 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:18.240561 31178 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:33629
I20260620 01:52:18.240646 31305 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:33629 every 8 connection(s)
I20260620 01:52:18.242976 31178 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:52:18.250190 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 31178
I20260620 01:52:18.277212 31315 sys_catalog.cc:455] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:18.279891 31315 sys_catalog.cc:458] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: This master's current role is: LEADER
I20260620 01:52:18.281023 31307 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40603
I20260620 01:52:18.281186 31306 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46463
I20260620 01:52:18.281554 31307 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:18.281561 31306 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:18.283303 31307 heartbeater.cc:507] Master 127.20.1.190:40603 requested a full tablet report, sending...
I20260620 01:52:18.283331 31306 heartbeater.cc:507] Master 127.20.1.188:46463 requested a full tablet report, sending...
I20260620 01:52:18.285434 31308 heartbeater.cc:344] Connected to a master server at 127.20.1.189:39115
I20260620 01:52:18.285722 31308 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:18.286542 31308 heartbeater.cc:507] Master 127.20.1.189:39115 requested a full tablet report, sending...
I20260620 01:52:18.286548 30567 ts_manager.cc:194] Re-registered known tserver with Master: 21330062fe2642cba2c0c86f7b17d5ed (127.20.1.129:33629)
I20260620 01:52:18.288430 30495 ts_manager.cc:194] Re-registered known tserver with Master: 21330062fe2642cba2c0c86f7b17d5ed (127.20.1.129:33629)
I20260620 01:52:18.288404 31118 ts_manager.cc:194] Registered new tserver with Master: 21330062fe2642cba2c0c86f7b17d5ed (127.20.1.129:33629)
I20260620 01:52:18.289608 30495 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:53865
I20260620 01:52:18.319072 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 31178
I20260620 01:52:18.338568 31301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:18.440627 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 31178
I20260620 01:52:18.456812 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:33629
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--webserver_port=43565
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:36667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
W20260620 01:52:18.933480 31322 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:18.933882 31322 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:18.934012 31322 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:52:18.934092 31322 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:18.943766 31322 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:18.944046 31322 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:52:18.957602 31322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:33629
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=43565
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:18.959839 31322 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:18.963284 31322 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:18.978019 31329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:18.977991 31331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:18.978243 31328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:18.980158 31322 server_base.cc:1061] running on GCE node
I20260620 01:52:18.981127 31322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:18.982607 31322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:18.984084 31322 hybrid_clock.cc:648] HybridClock initialized: now 1781920338983942 us; error 108 us; skew 500 ppm
I20260620 01:52:18.984529 31322 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:18.990279 31322 webserver.cc:533] Webserver started at http://127.20.1.129:43565/ using document root <none> and password file <none>
I20260620 01:52:18.991372 31322 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:18.991549 31322 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:19.000257 31322 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:52:19.004899 31337 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:19.007612 31322 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.002s
I20260620 01:52:19.007900 31322 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "21330062fe2642cba2c0c86f7b17d5ed"
format_stamp: "Formatted at 2026-06-20 01:52:14 on dist-test-slave-wg14"
I20260620 01:52:19.009054 31322 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:19.046243 31322 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:19.047605 31322 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:19.047937 31322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:19.049135 31322 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:52:19.051613 31322 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:52:19.051767 31322 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:19.051894 31322 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:52:19.051968 31322 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:52:19.052520 31322 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:52:19.052628 31322 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:52:19.052743 31322 fault_injection.cc:59] injecting fault for kudu (pid 31322): FLAGS_fault_before_start (process will exit)
I20260620 01:52:19.071204 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:33629
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--webserver_port=43565
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:36667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260620 01:52:19.569157 31345 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:19.569572 31345 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:19.569679 31345 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:19.581835 31345 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:19.582150 31345 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:52:19.598551 31345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:33629
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=43565
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:19.601481 31345 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:19.604396 31345 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:19.616947 31351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:19.616950 31350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:19.618685 31353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:19.619669 31345 server_base.cc:1061] running on GCE node
I20260620 01:52:19.620721 31345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:19.622046 31345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:19.623571 31345 hybrid_clock.cc:648] HybridClock initialized: now 1781920339623402 us; error 58 us; skew 500 ppm
I20260620 01:52:19.624040 31345 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:19.627321 31345 webserver.cc:533] Webserver started at http://127.20.1.129:43565/ using document root <none> and password file <none>
I20260620 01:52:19.628396 31345 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:19.628597 31345 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:19.636773 31345 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:52:19.642804 31359 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:19.645412 31345 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:52:19.645725 31345 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "21330062fe2642cba2c0c86f7b17d5ed"
format_stamp: "Formatted at 2026-06-20 01:52:14 on dist-test-slave-wg14"
I20260620 01:52:19.646857 31345 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:19.680704 31345 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:19.682142 31345 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:19.682476 31345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:19.683753 31345 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:52:19.685976 31345 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:52:19.686080 31345 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:19.686182 31345 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:52:19.686220 31345 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:19.736801 31345 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:33629
I20260620 01:52:19.736989 31471 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:33629 every 8 connection(s)
I20260620 01:52:19.738883 31345 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:52:19.746578 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 31345
I20260620 01:52:19.747100 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 31345
I20260620 01:52:19.783561 31472 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46463
I20260620 01:52:19.784210 31472 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:19.785092 31467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:19.784250 31473 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40603
I20260620 01:52:19.785631 31473 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:19.888612 31474 heartbeater.cc:344] Connected to a master server at 127.20.1.189:39115
I20260620 01:52:19.889048 31474 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:19.890594 31473 heartbeater.cc:507] Master 127.20.1.190:40603 requested a full tablet report, sending...
I20260620 01:52:19.891100 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 31345
I20260620 01:52:19.908881 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 30801
I20260620 01:52:19.921882 30924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:20.025872 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 30801
I20260620 01:52:20.042218 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 30939
I20260620 01:52:20.052668 31061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:20.157315 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 30939
I20260620 01:52:20.175180 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 31077
I20260620 01:52:20.185081 31148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:20.309473 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 31077
I20260620 01:52:20.338356 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 30461
I20260620 01:52:20.340111 30525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:20.470762 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 30461
I20260620 01:52:20.495317 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 30532
I20260620 01:52:20.497328 30597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:20.626668 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 30532
Shutting down hive metastore.
2026-06-20T01:52:21Z chronyd exiting
2026-06-20T01:52:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:52:21Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/security.properties
2026-06-20 01:52:22: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260620 01:52:32.818638 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40603
--webserver_interface=127.20.1.190
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463 with env {}
W20260620 01:52:33.282486 31677 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:33.283206 31677 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:33.283416 31677 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:33.294551 31677 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:52:33.294685 31677 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:33.294749 31677 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:52:33.294793 31677 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:52:33.308331 31677 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:33.310808 31677 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:33.313388 31677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:33.324774 31683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:33.324728 31682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:33.325801 31685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:33.326241 31677 server_base.cc:1061] running on GCE node
I20260620 01:52:33.327461 31677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:33.329444 31677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:33.330747 31677 hybrid_clock.cc:648] HybridClock initialized: now 1781920353330706 us; error 58 us; skew 500 ppm
I20260620 01:52:33.331290 31677 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:33.334517 31677 webserver.cc:533] Webserver started at http://127.20.1.190:39743/ using document root <none> and password file <none>
I20260620 01:52:33.335490 31677 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:33.335625 31677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:33.345263 31677 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 01:52:33.349349 31691 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:33.352123 31677 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.004s
I20260620 01:52:33.352438 31677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "a15143148e7b4afa83dc0ae5772ad3b3"
format_stamp: "Formatted at 2026-06-20 01:52:12 on dist-test-slave-wg14"
I20260620 01:52:33.353790 31677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:33.376511 31677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:33.377800 31677 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:33.378324 31677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:33.408141 31677 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:40603
I20260620 01:52:33.408155 31742 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:40603 every 8 connection(s)
I20260620 01:52:33.410281 31677 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:52:33.419183 31743 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:52:33.420418 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 31677
I20260620 01:52:33.420647 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:52:33.421876 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:39115
--webserver_interface=127.20.1.189
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463 with env {}
I20260620 01:52:33.424072 31743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: Bootstrap starting.
I20260620 01:52:33.447052 31743 log.cc:826] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: Log is configured to *not* fsync() on all Append() calls
I20260620 01:52:33.462134 31743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:52:33.463687 31743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: Bootstrap complete.
I20260620 01:52:33.475507 31743 raft_consensus.cc:359] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:33.476729 31743 raft_consensus.cc:740] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15143148e7b4afa83dc0ae5772ad3b3, State: Initialized, Role: FOLLOWER
I20260620 01:52:33.477783 31743 consensus_queue.cc:260] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:33.479180 31747 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:33.479633 31747 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:33.480703 31743 sys_catalog.cc:565] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:52:33.496732 31758 catalog_manager.cc:1294] Loaded cluster ID: 368ec3da3f0d467c970a416c3fb7fa92
I20260620 01:52:33.496943 31758 catalog_manager.cc:1587] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: loading cluster ID for follower catalog manager: success
I20260620 01:52:33.503232 31758 catalog_manager.cc:1609] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: acquiring CA information for follower catalog manager: success
I20260620 01:52:33.508335 31758 catalog_manager.cc:1637] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:52:33.527218 31677 master_runner.cc:187] Error getting master registration for 127.20.1.189:39115: Network error: Client connection negotiation failed: client connection to 127.20.1.189:39115: connect: Connection refused (error 111),
I20260620 01:52:33.534133 31677 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:52:33.910447 31745 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:33.910877 31745 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:33.911015 31745 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:33.921538 31745 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:52:33.921712 31745 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:33.921808 31745 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:52:33.921881 31745 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:52:33.936316 31745 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:39115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:33.938277 31745 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:33.940625 31745 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:33.952075 31766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:33.952172 31767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:33.955205 31769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:33.955868 31745 server_base.cc:1061] running on GCE node
I20260620 01:52:33.957038 31745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:33.959390 31745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:33.960798 31745 hybrid_clock.cc:648] HybridClock initialized: now 1781920353960711 us; error 61 us; skew 500 ppm
I20260620 01:52:33.961611 31745 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:33.965576 31745 webserver.cc:533] Webserver started at http://127.20.1.189:35915/ using document root <none> and password file <none>
I20260620 01:52:33.966926 31745 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:33.967118 31745 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:33.976238 31745 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 01:52:33.980430 31775 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:33.983206 31745 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:52:33.983518 31745 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "e887a75e4e8047708198e5dcc9bc2b6b"
format_stamp: "Formatted at 2026-06-20 01:52:12 on dist-test-slave-wg14"
I20260620 01:52:33.984581 31745 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:34.017088 31745 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:34.018169 31745 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:34.018553 31745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:34.048255 31826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:39115 every 8 connection(s)
I20260620 01:52:34.048247 31745 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:39115
I20260620 01:52:34.050427 31745 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:52:34.060151 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 31745
I20260620 01:52:34.060411 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:52:34.062119 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:46463
--webserver_interface=127.20.1.188
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463 with env {}
I20260620 01:52:34.062668 31827 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:52:34.066879 31827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b: Bootstrap starting.
I20260620 01:52:34.089985 31827 log.cc:826] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b: Log is configured to *not* fsync() on all Append() calls
I20260620 01:52:34.107075 31827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:52:34.107762 31827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b: Bootstrap complete.
I20260620 01:52:34.115397 31827 raft_consensus.cc:359] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:34.116231 31827 raft_consensus.cc:740] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e887a75e4e8047708198e5dcc9bc2b6b, State: Initialized, Role: FOLLOWER
I20260620 01:52:34.117142 31827 consensus_queue.cc:260] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:34.118157 31831 sys_catalog.cc:455] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:34.118866 31831 sys_catalog.cc:458] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:34.120292 31827 sys_catalog.cc:565] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:52:34.135938 31842 catalog_manager.cc:1294] Loaded cluster ID: 368ec3da3f0d467c970a416c3fb7fa92
I20260620 01:52:34.136439 31842 catalog_manager.cc:1587] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b: loading cluster ID for follower catalog manager: success
I20260620 01:52:34.142746 31842 catalog_manager.cc:1609] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b: acquiring CA information for follower catalog manager: success
I20260620 01:52:34.146373 31842 catalog_manager.cc:1637] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:52:34.198408 31745 master_runner.cc:187] Error getting master registration for 127.20.1.188:46463: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46463: connect: Connection refused (error 111),
I20260620 01:52:34.206183 31745 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:52:34.563758 31829 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:34.564256 31829 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:34.564527 31829 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:34.575744 31829 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:52:34.575892 31829 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:34.575968 31829 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:52:34.576030 31829 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:52:34.589777 31829 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:46463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:34.592340 31829 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:34.594710 31829 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:34.606923 31851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:34.607017 31852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:34.609175 31854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:34.610193 31829 server_base.cc:1061] running on GCE node
I20260620 01:52:34.611573 31829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:34.616031 31829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:34.617506 31829 hybrid_clock.cc:648] HybridClock initialized: now 1781920354617405 us; error 96 us; skew 500 ppm
I20260620 01:52:34.618083 31829 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:34.622298 31829 webserver.cc:533] Webserver started at http://127.20.1.188:44519/ using document root <none> and password file <none>
I20260620 01:52:34.623716 31829 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:34.623927 31829 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:34.634452 31829 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.005s sys 0.004s
I20260620 01:52:34.642228 31860 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:34.644786 31829 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260620 01:52:34.645112 31829 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "5a3fba60e67d4472940e013e0b864158"
format_stamp: "Formatted at 2026-06-20 01:52:13 on dist-test-slave-wg14"
I20260620 01:52:34.646405 31829 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:34.701784 31829 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:34.703219 31829 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:34.703668 31829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:34.745114 31829 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:46463
I20260620 01:52:34.745114 31911 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:46463 every 8 connection(s)
I20260620 01:52:34.747330 31829 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:52:34.755817 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 31829
I20260620 01:52:34.756064 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:52:34.757797 31912 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:52:34.763659 31912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158: Bootstrap starting.
I20260620 01:52:34.788383 31912 log.cc:826] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158: Log is configured to *not* fsync() on all Append() calls
I20260620 01:52:34.804491 31912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:52:34.805218 31912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158: Bootstrap complete.
I20260620 01:52:34.813549 31912 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:34.814575 31912 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3fba60e67d4472940e013e0b864158, State: Initialized, Role: FOLLOWER
I20260620 01:52:34.815866 31912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:34.817814 31919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:34.818137 31919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:34.818997 31912 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:52:34.839829 31930 catalog_manager.cc:1294] Loaded cluster ID: 368ec3da3f0d467c970a416c3fb7fa92
I20260620 01:52:34.840394 31930 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158: loading cluster ID for follower catalog manager: success
I20260620 01:52:34.846647 31930 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158: acquiring CA information for follower catalog manager: success
I20260620 01:52:34.851007 31930 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:52:34.874904 31829 master_runner.cc:264] No leader master found from master 5a3fba60e67d4472940e013e0b864158
I20260620 01:52:34.875083 31829 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:52:35.184340 31933 raft_consensus.cc:493] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:52:35.184916 31933 raft_consensus.cc:515] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:35.188261 31933 leader_election.cc:290] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e887a75e4e8047708198e5dcc9bc2b6b (127.20.1.189:39115), 5a3fba60e67d4472940e013e0b864158 (127.20.1.188:46463)
I20260620 01:52:35.198341 31802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" is_pre_election: true
I20260620 01:52:35.198555 31887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5a3fba60e67d4472940e013e0b864158" is_pre_election: true
I20260620 01:52:35.198968 31802 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a15143148e7b4afa83dc0ae5772ad3b3 in term 1.
I20260620 01:52:35.199349 31887 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a15143148e7b4afa83dc0ae5772ad3b3 in term 1.
I20260620 01:52:35.200343 31692 leader_election.cc:304] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15143148e7b4afa83dc0ae5772ad3b3, e887a75e4e8047708198e5dcc9bc2b6b; no voters:
I20260620 01:52:35.201062 31933 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:52:35.201335 31933 raft_consensus.cc:493] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:52:35.201462 31933 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:52:35.209836 31933 raft_consensus.cc:515] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:35.211738 31802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e887a75e4e8047708198e5dcc9bc2b6b"
I20260620 01:52:35.212040 31802 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:52:35.212055 31887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5a3fba60e67d4472940e013e0b864158"
I20260620 01:52:35.212467 31887 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:52:35.215327 31933 leader_election.cc:290] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [CANDIDATE]: Term 2 election: Requested vote from peers e887a75e4e8047708198e5dcc9bc2b6b (127.20.1.189:39115), 5a3fba60e67d4472940e013e0b864158 (127.20.1.188:46463)
I20260620 01:52:35.220732 31887 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a15143148e7b4afa83dc0ae5772ad3b3 in term 2.
I20260620 01:52:35.221939 31693 leader_election.cc:304] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a3fba60e67d4472940e013e0b864158, a15143148e7b4afa83dc0ae5772ad3b3; no voters:
I20260620 01:52:35.222087 31802 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a15143148e7b4afa83dc0ae5772ad3b3 in term 2.
I20260620 01:52:35.222620 31933 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 01:52:35.223351 31933 raft_consensus.cc:697] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 2 LEADER]: Becoming Leader. State: Replica: a15143148e7b4afa83dc0ae5772ad3b3, State: Running, Role: LEADER
I20260620 01:52:35.224048 31933 consensus_queue.cc:237] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:35.228089 31936 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a15143148e7b4afa83dc0ae5772ad3b3. Latest consensus state: current_term: 2 leader_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:35.228549 31936 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: This master's current role is: LEADER
I20260620 01:52:35.229564 31938 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:52:35.232455 31938 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:52:35.233702 31938 catalog_manager.cc:1294] Loaded cluster ID: 368ec3da3f0d467c970a416c3fb7fa92
I20260620 01:52:35.233835 31938 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:52:35.235517 31938 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:52:35.236992 31938 catalog_manager.cc:6080] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: Loaded TSK: 0
I20260620 01:52:35.238457 31938 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:52:35.245568 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--builtin_ntp_servers=127.20.1.148:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:52:35.615216 31887 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 2 FOLLOWER]: Refusing update from remote peer a15143148e7b4afa83dc0ae5772ad3b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:52:35.617332 31936 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:52:35.624423 31940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a15143148e7b4afa83dc0ae5772ad3b3. Latest consensus state: current_term: 2 leader_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:35.624783 31940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:35.627334 31936 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:35.627655 31936 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: This master's current role is: LEADER
I20260620 01:52:35.628657 31940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:35.628904 31940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:35.628872 31802 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 2 FOLLOWER]: Refusing update from remote peer a15143148e7b4afa83dc0ae5772ad3b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:52:35.630059 31933 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:52:35.639020 31947 sys_catalog.cc:455] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:35.639598 31947 sys_catalog.cc:458] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:35.640156 31944 sys_catalog.cc:455] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a15143148e7b4afa83dc0ae5772ad3b3. Latest consensus state: current_term: 2 leader_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:35.640444 31944 sys_catalog.cc:458] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:52:35.723829 31939 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:35.724220 31939 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:35.724341 31939 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:35.734160 31939 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:35.734354 31939 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:52:35.746589 31939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:35.748585 31939 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:35.750973 31939 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:35.766402 31952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:35.766579 31955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:35.766386 31953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:35.768110 31939 server_base.cc:1061] running on GCE node
I20260620 01:52:35.768873 31939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:35.770210 31939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:35.771765 31939 hybrid_clock.cc:648] HybridClock initialized: now 1781920355771644 us; error 71 us; skew 500 ppm
I20260620 01:52:35.772295 31939 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:35.775869 31939 webserver.cc:533] Webserver started at http://127.20.1.129:38583/ using document root <none> and password file <none>
I20260620 01:52:35.777014 31939 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:35.777197 31939 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:35.784637 31939 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 01:52:35.789826 31961 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:35.792011 31939 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:52:35.792275 31939 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "21330062fe2642cba2c0c86f7b17d5ed"
format_stamp: "Formatted at 2026-06-20 01:52:14 on dist-test-slave-wg14"
I20260620 01:52:35.793334 31939 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:35.812399 31939 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:35.813452 31939 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:35.813782 31939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:35.815002 31939 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:52:35.817549 31939 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:52:35.817703 31939 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:35.817853 31939 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:52:35.817909 31939 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:35.863274 31939 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:45367
I20260620 01:52:35.863320 32073 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:45367 every 8 connection(s)
I20260620 01:52:35.865348 31939 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:52:35.875456 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 31939
I20260620 01:52:35.875739 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:52:35.885380 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--builtin_ntp_servers=127.20.1.148:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:52:35.936391 32075 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40603
I20260620 01:52:35.937007 32075 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:35.938220 32075 heartbeater.cc:507] Master 127.20.1.190:40603 requested a full tablet report, sending...
I20260620 01:52:35.938282 32076 heartbeater.cc:344] Connected to a master server at 127.20.1.189:39115
I20260620 01:52:35.938673 32076 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:35.939316 32074 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46463
I20260620 01:52:35.939549 32074 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:35.939862 32076 heartbeater.cc:507] Master 127.20.1.189:39115 requested a full tablet report, sending...
I20260620 01:52:35.940196 32074 heartbeater.cc:507] Master 127.20.1.188:46463 requested a full tablet report, sending...
I20260620 01:52:35.942497 31708 ts_manager.cc:194] Registered new tserver with Master: 21330062fe2642cba2c0c86f7b17d5ed (127.20.1.129:45367)
I20260620 01:52:35.943419 31791 ts_manager.cc:194] Registered new tserver with Master: 21330062fe2642cba2c0c86f7b17d5ed (127.20.1.129:45367)
I20260620 01:52:35.945199 31708 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:51003
I20260620 01:52:35.947281 31877 ts_manager.cc:194] Registered new tserver with Master: 21330062fe2642cba2c0c86f7b17d5ed (127.20.1.129:45367)
I20260620 01:52:36.139366 31933 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:36.139755 31933 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: This master's current role is: LEADER
W20260620 01:52:36.394207 32084 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:36.394618 32084 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:36.394726 32084 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:36.405133 32084 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:36.405391 32084 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:52:36.424420 32084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:36.426795 32084 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:36.429214 32084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:36.443739 32093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:36.444391 32092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:36.444541 32095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:36.446350 32084 server_base.cc:1061] running on GCE node
I20260620 01:52:36.447625 32084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:36.448977 32084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:36.450507 32084 hybrid_clock.cc:648] HybridClock initialized: now 1781920356450259 us; error 103 us; skew 500 ppm
I20260620 01:52:36.451488 32084 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:36.454995 32084 webserver.cc:533] Webserver started at http://127.20.1.130:36479/ using document root <none> and password file <none>
I20260620 01:52:36.456245 32084 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:36.456440 32084 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:36.465085 32084 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 01:52:36.469628 32101 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:36.473245 32084 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260620 01:52:36.473763 32084 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "7f5ceedc91a6418497b6a16dff4ed5d8"
format_stamp: "Formatted at 2026-06-20 01:52:15 on dist-test-slave-wg14"
I20260620 01:52:36.475087 32084 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:36.507234 32084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:36.508570 32084 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:36.508934 32084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:36.510421 32084 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:52:36.513039 32084 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:52:36.513196 32084 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:36.513358 32084 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:52:36.513434 32084 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:36.567301 32084 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:34187
I20260620 01:52:36.567662 32213 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:34187 every 8 connection(s)
I20260620 01:52:36.569356 32084 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:52:36.570845 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 32084
I20260620 01:52:36.571378 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
I20260620 01:52:36.581004 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--builtin_ntp_servers=127.20.1.148:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:52:36.603147 32215 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40603
I20260620 01:52:36.603732 32215 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:36.604563 32214 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46463
I20260620 01:52:36.604784 32214 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:36.605247 32215 heartbeater.cc:507] Master 127.20.1.190:40603 requested a full tablet report, sending...
I20260620 01:52:36.605351 32214 heartbeater.cc:507] Master 127.20.1.188:46463 requested a full tablet report, sending...
I20260620 01:52:36.607693 31708 ts_manager.cc:194] Registered new tserver with Master: 7f5ceedc91a6418497b6a16dff4ed5d8 (127.20.1.130:34187)
I20260620 01:52:36.607820 31877 ts_manager.cc:194] Registered new tserver with Master: 7f5ceedc91a6418497b6a16dff4ed5d8 (127.20.1.130:34187)
I20260620 01:52:36.608413 32216 heartbeater.cc:344] Connected to a master server at 127.20.1.189:39115
I20260620 01:52:36.608640 32216 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:36.609231 32216 heartbeater.cc:507] Master 127.20.1.189:39115 requested a full tablet report, sending...
I20260620 01:52:36.609359 31708 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.130:58291
I20260620 01:52:36.610906 31791 ts_manager.cc:194] Registered new tserver with Master: 7f5ceedc91a6418497b6a16dff4ed5d8 (127.20.1.130:34187)
I20260620 01:52:36.948732 32075 heartbeater.cc:499] Master 127.20.1.190:40603 was elected leader, sending a full tablet report...
W20260620 01:52:37.132630 32224 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:37.133083 32224 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:37.133270 32224 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:37.144589 32224 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:37.144853 32224 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:52:37.157789 32224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:37.160151 32224 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:37.163458 32224 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:37.178622 32232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:37.178586 32230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:37.179183 32229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:37.180079 32224 server_base.cc:1061] running on GCE node
I20260620 01:52:37.180975 32224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:37.182294 32224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:37.183558 32224 hybrid_clock.cc:648] HybridClock initialized: now 1781920357183481 us; error 75 us; skew 500 ppm
I20260620 01:52:37.183946 32224 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:37.187218 32224 webserver.cc:533] Webserver started at http://127.20.1.131:40965/ using document root <none> and password file <none>
I20260620 01:52:37.188495 32224 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:37.188683 32224 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:37.197305 32224 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:52:37.203385 32238 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:37.206084 32224 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260620 01:52:37.206377 32224 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "e1f9b3cf197948de83c943800b4c90ad"
format_stamp: "Formatted at 2026-06-20 01:52:16 on dist-test-slave-wg14"
I20260620 01:52:37.207767 32224 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:37.252199 32224 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:37.254014 32224 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:37.254894 32224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:37.256925 32224 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:52:37.260006 32224 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:52:37.260164 32224 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:37.260299 32224 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:52:37.260363 32224 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:37.308486 32224 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:42803
I20260620 01:52:37.308550 32350 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:42803 every 8 connection(s)
I20260620 01:52:37.310202 32224 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:52:37.316318 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 32224
I20260620 01:52:37.316603 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
I20260620 01:52:37.348942 32352 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40603
I20260620 01:52:37.349529 32352 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:37.349740 32351 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46463
I20260620 01:52:37.349910 32351 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:37.350399 32353 heartbeater.cc:344] Connected to a master server at 127.20.1.189:39115
I20260620 01:52:37.350848 32353 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:37.351174 32352 heartbeater.cc:507] Master 127.20.1.190:40603 requested a full tablet report, sending...
I20260620 01:52:37.351336 32351 heartbeater.cc:507] Master 127.20.1.188:46463 requested a full tablet report, sending...
I20260620 01:52:37.351887 32353 heartbeater.cc:507] Master 127.20.1.189:39115 requested a full tablet report, sending...
I20260620 01:52:37.354910 31877 ts_manager.cc:194] Registered new tserver with Master: e1f9b3cf197948de83c943800b4c90ad (127.20.1.131:42803)
I20260620 01:52:37.356143 31708 ts_manager.cc:194] Registered new tserver with Master: e1f9b3cf197948de83c943800b4c90ad (127.20.1.131:42803)
I20260620 01:52:37.357173 31791 ts_manager.cc:194] Registered new tserver with Master: e1f9b3cf197948de83c943800b4c90ad (127.20.1.131:42803)
I20260620 01:52:37.358194 31708 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.131:58855
I20260620 01:52:37.366689 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:52:37.383447 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:52:37.383688 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 31677
I20260620 01:52:37.385342 31738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:37.533191 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 31677
I20260620 01:52:37.558120 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40603
--webserver_interface=127.20.1.190
--webserver_port=39743
--builtin_ntp_servers=127.20.1.148:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463 with env {}
I20260620 01:52:37.614470 32215 heartbeater.cc:499] Master 127.20.1.190:40603 was elected leader, sending a full tablet report...
W20260620 01:52:37.617439 32215 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:40603 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:40603: connect: Connection refused (error 111)
W20260620 01:52:37.953557 32075 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:40603 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:40603: connect: Connection refused (error 111)
W20260620 01:52:38.057315 32362 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:38.057767 32362 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:38.057895 32362 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:38.069722 32362 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:52:38.070160 32362 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:38.070350 32362 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:52:38.070423 32362 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:52:38.083546 32362 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=39743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:38.085958 32362 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:38.089934 32362 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:38.104983 32369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:38.105304 32370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:38.105324 32372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:38.106726 32362 server_base.cc:1061] running on GCE node
I20260620 01:52:38.108654 32362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:38.111770 32362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:38.113131 32362 hybrid_clock.cc:648] HybridClock initialized: now 1781920358113070 us; error 54 us; skew 500 ppm
I20260620 01:52:38.113700 32362 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:38.117365 32362 webserver.cc:533] Webserver started at http://127.20.1.190:39743/ using document root <none> and password file <none>
I20260620 01:52:38.118439 32362 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:38.118600 32362 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:38.128089 32362 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.007s sys 0.001s
I20260620 01:52:38.132676 32378 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:38.135396 32362 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:52:38.135674 32362 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "a15143148e7b4afa83dc0ae5772ad3b3"
format_stamp: "Formatted at 2026-06-20 01:52:12 on dist-test-slave-wg14"
I20260620 01:52:38.136754 32362 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:38.167296 32362 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:38.168524 32362 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:38.168846 32362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:38.197896 32362 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:40603
I20260620 01:52:38.197909 32429 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:40603 every 8 connection(s)
I20260620 01:52:38.201251 32362 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:52:38.207960 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 32362
I20260620 01:52:38.208348 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 31939
I20260620 01:52:38.212867 32430 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:52:38.222133 32430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: Bootstrap starting.
I20260620 01:52:38.225356 32069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:38.252661 32430 log.cc:826] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: Log is configured to *not* fsync() on all Append() calls
I20260620 01:52:38.269963 32430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:52:38.270656 32430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: Bootstrap complete.
I20260620 01:52:38.279292 32430 raft_consensus.cc:359] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:38.280763 32430 raft_consensus.cc:740] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15143148e7b4afa83dc0ae5772ad3b3, State: Initialized, Role: FOLLOWER
I20260620 01:52:38.281793 32430 consensus_queue.cc:260] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:38.282984 32436 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:38.283390 32436 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:38.284545 32430 sys_catalog.cc:565] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:52:38.324636 32447 catalog_manager.cc:1294] Loaded cluster ID: 368ec3da3f0d467c970a416c3fb7fa92
I20260620 01:52:38.324925 32447 catalog_manager.cc:1587] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: loading cluster ID for follower catalog manager: success
I20260620 01:52:38.331524 32447 catalog_manager.cc:1609] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: acquiring CA information for follower catalog manager: success
I20260620 01:52:38.335745 32447 catalog_manager.cc:1637] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:52:38.355572 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 31939
I20260620 01:52:38.362495 32352 heartbeater.cc:499] Master 127.20.1.190:40603 was elected leader, sending a full tablet report...
I20260620 01:52:38.383754 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:45367
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--webserver_port=38583
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:52:38.390367 32395 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e1f9b3cf197948de83c943800b4c90ad" instance_seqno: 1781920357297018) as {username='slave'} at 127.20.1.131:42387; Asking this server to re-register.
I20260620 01:52:38.391736 32352 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:38.391748 32362 master_runner.cc:264] No leader master found from master a15143148e7b4afa83dc0ae5772ad3b3
I20260620 01:52:38.391916 32362 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:52:38.394572 32395 ts_manager.cc:194] Registered new tserver with Master: e1f9b3cf197948de83c943800b4c90ad (127.20.1.131:42803)
I20260620 01:52:38.486387 32455 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a15143148e7b4afa83dc0ae5772ad3b3)
I20260620 01:52:38.486716 32455 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:38.489599 32455 leader_election.cc:290] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a15143148e7b4afa83dc0ae5772ad3b3 (127.20.1.190:40603), e887a75e4e8047708198e5dcc9bc2b6b (127.20.1.189:39115)
I20260620 01:52:38.491591 31802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a3fba60e67d4472940e013e0b864158" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" is_pre_election: true
I20260620 01:52:38.505344 32405 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a3fba60e67d4472940e013e0b864158" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" is_pre_election: true
I20260620 01:52:38.506256 32405 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a3fba60e67d4472940e013e0b864158 in term 2.
I20260620 01:52:38.507400 31863 leader_election.cc:304] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a3fba60e67d4472940e013e0b864158, a15143148e7b4afa83dc0ae5772ad3b3; no voters: e887a75e4e8047708198e5dcc9bc2b6b
I20260620 01:52:38.508049 32455 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 01:52:38.508224 32455 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a15143148e7b4afa83dc0ae5772ad3b3)
I20260620 01:52:38.508342 32455 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:52:38.511664 32455 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:38.513857 31802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a3fba60e67d4472940e013e0b864158" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e887a75e4e8047708198e5dcc9bc2b6b"
I20260620 01:52:38.513885 32405 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a3fba60e67d4472940e013e0b864158" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a15143148e7b4afa83dc0ae5772ad3b3"
I20260620 01:52:38.514444 32405 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:52:38.516362 32455 leader_election.cc:290] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [CANDIDATE]: Term 3 election: Requested vote from peers a15143148e7b4afa83dc0ae5772ad3b3 (127.20.1.190:40603), e887a75e4e8047708198e5dcc9bc2b6b (127.20.1.189:39115)
I20260620 01:52:38.524753 32405 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a3fba60e67d4472940e013e0b864158 in term 3.
I20260620 01:52:38.525784 31863 leader_election.cc:304] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a3fba60e67d4472940e013e0b864158, a15143148e7b4afa83dc0ae5772ad3b3; no voters: e887a75e4e8047708198e5dcc9bc2b6b
I20260620 01:52:38.526510 32455 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 01:52:38.527448 32455 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [term 3 LEADER]: Becoming Leader. State: Replica: 5a3fba60e67d4472940e013e0b864158, State: Running, Role: LEADER
I20260620 01:52:38.528307 32455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:38.531225 32457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a3fba60e67d4472940e013e0b864158. Latest consensus state: current_term: 3 leader_uuid: "5a3fba60e67d4472940e013e0b864158" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:38.531663 32457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: This master's current role is: LEADER
I20260620 01:52:38.532694 32459 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:52:38.537388 32459 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:52:38.539229 32459 catalog_manager.cc:1294] Loaded cluster ID: 368ec3da3f0d467c970a416c3fb7fa92
I20260620 01:52:38.539405 32459 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:52:38.541597 32459 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:52:38.543064 32459 catalog_manager.cc:6080] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158: Loaded TSK: 0
I20260620 01:52:38.544595 32459 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:52:38.637020 32215 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40603
I20260620 01:52:38.637470 32215 heartbeater.cc:499] Master 127.20.1.190:40603 was elected leader, sending a full tablet report...
I20260620 01:52:38.639629 32395 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7f5ceedc91a6418497b6a16dff4ed5d8" instance_seqno: 1781920356552059) as {username='slave'} at 127.20.1.130:38655; Asking this server to re-register.
I20260620 01:52:38.641265 32215 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:38.643913 32395 ts_manager.cc:194] Registered new tserver with Master: 7f5ceedc91a6418497b6a16dff4ed5d8 (127.20.1.130:34187)
I20260620 01:52:38.645630 32461 raft_consensus.cc:493] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a15143148e7b4afa83dc0ae5772ad3b3)
I20260620 01:52:38.645969 32461 raft_consensus.cc:515] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } }
I20260620 01:52:38.648838 32461 leader_election.cc:290] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a15143148e7b4afa83dc0ae5772ad3b3 (127.20.1.190:40603), 5a3fba60e67d4472940e013e0b864158 (127.20.1.188:46463)
I20260620 01:52:38.674619 32405 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" is_pre_election: true
I20260620 01:52:38.675320 31887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5a3fba60e67d4472940e013e0b864158" is_pre_election: true
I20260620 01:52:38.676568 32405 raft_consensus.cc:2393] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e887a75e4e8047708198e5dcc9bc2b6b in current term 3: Already voted for candidate 5a3fba60e67d4472940e013e0b864158 in this term.
I20260620 01:52:38.679080 31778 leader_election.cc:304] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e887a75e4e8047708198e5dcc9bc2b6b; no voters: 5a3fba60e67d4472940e013e0b864158, a15143148e7b4afa83dc0ae5772ad3b3
I20260620 01:52:38.680042 32461 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:52:38.684199 32461 raft_consensus.cc:2749] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 01:52:38.950567 32405 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [term 3 FOLLOWER]: Refusing update from remote peer 5a3fba60e67d4472940e013e0b864158: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:52:38.952222 32457 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 01:52:38.959944 32464 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a3fba60e67d4472940e013e0b864158. Latest consensus state: current_term: 3 leader_uuid: "5a3fba60e67d4472940e013e0b864158" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:38.960731 32464 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:38.964315 31802 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [term 3 FOLLOWER]: Refusing update from remote peer 5a3fba60e67d4472940e013e0b864158: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:52:38.962926 32457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "5a3fba60e67d4472940e013e0b864158" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:38.965519 32457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: This master's current role is: LEADER
I20260620 01:52:38.965765 32457 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:52:38.976248 32464 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5a3fba60e67d4472940e013e0b864158" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:38.976599 32464 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15143148e7b4afa83dc0ae5772ad3b3 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:52:38.976696 32454 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:38.977242 32454 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:38.977453 32454 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260620 01:52:38.979542 32474 sys_catalog.cc:455] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5a3fba60e67d4472940e013e0b864158" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:38.979799 32461 sys_catalog.cc:455] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a3fba60e67d4472940e013e0b864158. Latest consensus state: current_term: 3 leader_uuid: "5a3fba60e67d4472940e013e0b864158" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:38.980119 32461 sys_catalog.cc:458] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:52:38.980372 32474 sys_catalog.cc:458] T 00000000000000000000000000000000 P e887a75e4e8047708198e5dcc9bc2b6b [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:52:38.989429 32454 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:38.989677 32454 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:52:39.008272 32454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:45367
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=38583
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:39.010284 32454 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:39.012701 32454 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:39.030946 32477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:39.031028 32476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:39.030956 32479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:39.032441 32454 server_base.cc:1061] running on GCE node
I20260620 01:52:39.033421 32454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:39.034899 32454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:39.036602 32454 hybrid_clock.cc:648] HybridClock initialized: now 1781920359036440 us; error 181 us; skew 500 ppm
I20260620 01:52:39.037122 32454 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:39.042742 32454 webserver.cc:533] Webserver started at http://127.20.1.129:38583/ using document root <none> and password file <none>
I20260620 01:52:39.044240 32454 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:39.044441 32454 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:39.055074 32454 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.006s sys 0.000s
I20260620 01:52:39.060261 32485 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:39.062804 32454 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20260620 01:52:39.063196 32454 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "21330062fe2642cba2c0c86f7b17d5ed"
format_stamp: "Formatted at 2026-06-20 01:52:14 on dist-test-slave-wg14"
I20260620 01:52:39.064328 32454 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:39.095964 32454 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:39.097766 32454 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:39.098204 32454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:39.100111 32454 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:52:39.103282 32454 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:52:39.103436 32454 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:39.103555 32454 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:52:39.103638 32454 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:39.151937 32454 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:45367
I20260620 01:52:39.152014 32597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:45367 every 8 connection(s)
I20260620 01:52:39.154397 32454 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:52:39.160146 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 32454
I20260620 01:52:39.191231 32599 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40603
I20260620 01:52:39.191921 32599 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:39.193737 32599 heartbeater.cc:507] Master 127.20.1.190:40603 requested a full tablet report, sending...
I20260620 01:52:39.197594 32395 ts_manager.cc:194] Registered new tserver with Master: 21330062fe2642cba2c0c86f7b17d5ed (127.20.1.129:45367)
I20260620 01:52:39.202838 32598 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46463
I20260620 01:52:39.203104 32598 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:39.203805 32598 heartbeater.cc:507] Master 127.20.1.188:46463 requested a full tablet report, sending...
I20260620 01:52:39.206331 31877 ts_manager.cc:194] Re-registered known tserver with Master: 21330062fe2642cba2c0c86f7b17d5ed (127.20.1.129:45367)
I20260620 01:52:39.209771 31877 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:34007
I20260620 01:52:39.215826 32600 heartbeater.cc:344] Connected to a master server at 127.20.1.189:39115
I20260620 01:52:39.216096 32600 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:39.216585 32600 heartbeater.cc:507] Master 127.20.1.189:39115 requested a full tablet report, sending...
I20260620 01:52:39.218590 31792 ts_manager.cc:194] Re-registered known tserver with Master: 21330062fe2642cba2c0c86f7b17d5ed (127.20.1.129:45367)
I20260620 01:52:39.339349 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 32454
I20260620 01:52:39.364372 32593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:39.434782 32455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "5a3fba60e67d4472940e013e0b864158" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15143148e7b4afa83dc0ae5772ad3b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40603 } } peers { permanent_uuid: "e887a75e4e8047708198e5dcc9bc2b6b" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 39115 } } peers { permanent_uuid: "5a3fba60e67d4472940e013e0b864158" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46463 } } }
I20260620 01:52:39.435259 32455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3fba60e67d4472940e013e0b864158 [sys.catalog]: This master's current role is: LEADER
I20260620 01:52:39.472657 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 32454
I20260620 01:52:39.489914 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:45367
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--webserver_port=38583
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
I20260620 01:52:39.622887 32214 heartbeater.cc:499] Master 127.20.1.188:46463 was elected leader, sending a full tablet report...
W20260620 01:52:39.937893 32610 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:39.938503 32610 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:39.938694 32610 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:52:39.938791 32610 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:39.951494 32610 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:39.951830 32610 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:52:39.965044 32610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:45367
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=38583
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:39.967020 32610 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:39.970180 32610 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:39.985479 32615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:39.985733 32616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:39.986366 32618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:39.988341 32610 server_base.cc:1061] running on GCE node
I20260620 01:52:39.989255 32610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:39.991110 32610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:39.992712 32610 hybrid_clock.cc:648] HybridClock initialized: now 1781920359992619 us; error 69 us; skew 500 ppm
I20260620 01:52:39.993216 32610 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:39.996541 32610 webserver.cc:533] Webserver started at http://127.20.1.129:38583/ using document root <none> and password file <none>
I20260620 01:52:39.997634 32610 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:39.997792 32610 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:40.005796 32610 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:52:40.011027 32624 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:40.013599 32610 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260620 01:52:40.013897 32610 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "21330062fe2642cba2c0c86f7b17d5ed"
format_stamp: "Formatted at 2026-06-20 01:52:14 on dist-test-slave-wg14"
I20260620 01:52:40.015230 32610 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:40.043741 32610 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:40.045094 32610 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:40.045464 32610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:40.046818 32610 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:52:40.050087 32610 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:52:40.050264 32610 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:40.050539 32610 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:52:40.050630 32610 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:52:40.051800 32610 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:52:40.051930 32610 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:52:40.052000 32610 fault_injection.cc:59] injecting fault for kudu (pid 32610): FLAGS_fault_before_start (process will exit)
I20260620 01:52:40.070122 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:45367
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--webserver_port=38583
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:52:40.363238 32351 heartbeater.cc:499] Master 127.20.1.188:46463 was elected leader, sending a full tablet report...
W20260620 01:52:40.516041 32632 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:52:40.516505 32632 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:52:40.516635 32632 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:52:40.527319 32632 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:52:40.527581 32632 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:52:40.540391 32632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:45367
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=38583
--tserver_master_addrs=127.20.1.190:40603,127.20.1.189:39115,127.20.1.188:46463
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:52:40.542081 32632 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:52:40.544397 32632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:52:40.559960 32637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:40.559938 32638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:52:40.560884 32640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:52:40.562435 32632 server_base.cc:1061] running on GCE node
I20260620 01:52:40.563695 32632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:52:40.566444 32632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:52:40.568020 32632 hybrid_clock.cc:648] HybridClock initialized: now 1781920360567927 us; error 74 us; skew 500 ppm
I20260620 01:52:40.568578 32632 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:52:40.572396 32632 webserver.cc:533] Webserver started at http://127.20.1.129:38583/ using document root <none> and password file <none>
I20260620 01:52:40.573868 32632 fs_manager.cc:362] Metadata directory not provided
I20260620 01:52:40.574419 32632 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:52:40.585878 32632 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.008s sys 0.000s
I20260620 01:52:40.590799 32646 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:40.593900 32632 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:52:40.594192 32632 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "21330062fe2642cba2c0c86f7b17d5ed"
format_stamp: "Formatted at 2026-06-20 01:52:14 on dist-test-slave-wg14"
I20260620 01:52:40.595842 32632 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:52:40.647313 32632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:52:40.648727 32632 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:52:40.649094 32632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:52:40.650440 32632 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:52:40.653324 32632 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:52:40.653524 32632 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:52:40.653697 32632 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:52:40.653818 32632 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 01:52:40.705905 32632 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:45367
I20260620 01:52:40.706020 32758 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:45367 every 8 connection(s)
I20260620 01:52:40.708341 32632 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:52:40.712219 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 32632
I20260620 01:52:40.712667 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 32632
I20260620 01:52:40.755842 32759 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46463
I20260620 01:52:40.755879 32760 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40603
I20260620 01:52:40.756103 32761 heartbeater.cc:344] Connected to a master server at 127.20.1.189:39115
I20260620 01:52:40.756553 32761 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:40.756554 32759 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:40.756554 32760 heartbeater.cc:461] Registering TS with master...
I20260620 01:52:40.758486 32759 heartbeater.cc:507] Master 127.20.1.188:46463 requested a full tablet report, sending...
I20260620 01:52:40.758708 32754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:40.758553 32761 heartbeater.cc:507] Master 127.20.1.189:39115 requested a full tablet report, sending...
I20260620 01:52:40.761308 31877 ts_manager.cc:194] Re-registered known tserver with Master: 21330062fe2642cba2c0c86f7b17d5ed (127.20.1.129:45367)
I20260620 01:52:40.761489 31792 ts_manager.cc:194] Re-registered known tserver with Master: 21330062fe2642cba2c0c86f7b17d5ed (127.20.1.129:45367)
I20260620 01:52:40.762943 31877 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:51979
I20260620 01:52:40.864440 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 32632
I20260620 01:52:40.879508 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 32084
I20260620 01:52:40.891654 32209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:40.997535 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 32084
I20260620 01:52:41.015529 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 32224
I20260620 01:52:41.033553 32346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:41.153854 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 32224
I20260620 01:52:41.169245 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 32362
I20260620 01:52:41.177968 32425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:41.308528 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 32362
I20260620 01:52:41.329710 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 31745
I20260620 01:52:41.331164 31822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:41.464540 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 31745
W20260620 01:52:41.483872 31861 connection.cc:573] client connection to 127.20.1.189:39115 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:52:41.484180 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 31829
W20260620 01:52:41.484251 31861 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:52:41.485980 31907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:52:41.615003 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 31829
Shutting down hive metastore.
2026-06-20T01:52:42Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/4 (53399 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/5
2026-06-20T01:52:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:52:42Z Disabled control of system clock
2026-06-20T01:52:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:52:42Z Disabled control of system clock
2026-06-20T01:52:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:52:42Z Disabled control of system clock
2026-06-20T01:52:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:52:42Z Disabled control of system clock
2026-06-20T01:52:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:52:42Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/security.properties
2026-06-20 01:52:52: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260620 01:53:03.471426 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40263
--webserver_interface=127.20.1.190
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299 with env {}
W20260620 01:53:03.888077 613 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:03.888594 613 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:03.888756 613 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:03.899545 613 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:03.899694 613 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:03.899780 613 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:03.899852 613 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:03.912631 613 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:03.914811 613 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:03.917583 613 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:03.930238 619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:03.930233 618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:03.932757 621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:03.933173 613 server_base.cc:1061] running on GCE node
I20260620 01:53:03.934226 613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:03.936910 613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:03.940519 613 hybrid_clock.cc:648] HybridClock initialized: now 1781920383940300 us; error 131 us; skew 500 ppm
I20260620 01:53:03.941089 613 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:03.945103 613 webserver.cc:533] Webserver started at http://127.20.1.190:36149/ using document root <none> and password file <none>
I20260620 01:53:03.946501 613 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:03.946682 613 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:03.947439 613 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:53:03.951038 613 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "025e1c47b0614610ad9da9addf02bdaf"
format_stamp: "Formatted at 2026-06-20 01:53:03 on dist-test-slave-wg14"
I20260620 01:53:03.952253 613 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "025e1c47b0614610ad9da9addf02bdaf"
format_stamp: "Formatted at 2026-06-20 01:53:03 on dist-test-slave-wg14"
I20260620 01:53:03.961068 613 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260620 01:53:03.967164 627 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:03.969630 613 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260620 01:53:03.969916 613 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "025e1c47b0614610ad9da9addf02bdaf"
format_stamp: "Formatted at 2026-06-20 01:53:03 on dist-test-slave-wg14"
I20260620 01:53:03.970216 613 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:03.989533 613 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:03.991101 613 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:03.991528 613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:04.022267 613 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:40263
I20260620 01:53:04.022267 678 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:40263 every 8 connection(s)
I20260620 01:53:04.024456 613 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:53:04.030186 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 613
I20260620 01:53:04.030257 679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:53:04.030405 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:53:04.031622 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:36607
--webserver_interface=127.20.1.189
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299 with env {}
I20260620 01:53:04.038656 679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:53:04.067337 679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:53:04.069427 631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36607: connect: Connection refused (error 111)
W20260620 01:53:04.072628 679 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36607: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36607: connect: Connection refused (error 111)
I20260620 01:53:04.122083 679 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } attempt: 1
W20260620 01:53:04.124734 679 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36607: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36607: connect: Connection refused (error 111)
I20260620 01:53:04.193275 679 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } attempt: 2
W20260620 01:53:04.195722 679 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36607: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36607: connect: Connection refused (error 111)
I20260620 01:53:04.287086 679 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } attempt: 3
W20260620 01:53:04.289645 679 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36607: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36607: connect: Connection refused (error 111)
I20260620 01:53:04.433063 679 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } attempt: 4
W20260620 01:53:04.435698 679 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36607: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36607: connect: Connection refused (error 111)
W20260620 01:53:04.524288 681 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:04.524798 681 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:04.524914 681 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:04.534582 681 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:04.534720 681 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:04.534780 681 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:04.534821 681 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:04.549412 681 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:36607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:04.551373 681 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:04.553649 681 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:04.565601 690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:04.565650 691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:04.566617 693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:04.568010 681 server_base.cc:1061] running on GCE node
I20260620 01:53:04.569222 681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:04.571714 681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:04.575591 681 hybrid_clock.cc:648] HybridClock initialized: now 1781920384575268 us; error 118 us; skew 500 ppm
I20260620 01:53:04.576221 681 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:04.579952 681 webserver.cc:533] Webserver started at http://127.20.1.189:36531/ using document root <none> and password file <none>
I20260620 01:53:04.580945 681 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:04.581079 681 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:04.581489 681 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:53:04.584440 681 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "b171c7ed0a74499abcaa971b8f349a97"
format_stamp: "Formatted at 2026-06-20 01:53:04 on dist-test-slave-wg14"
I20260620 01:53:04.585671 681 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "b171c7ed0a74499abcaa971b8f349a97"
format_stamp: "Formatted at 2026-06-20 01:53:04 on dist-test-slave-wg14"
I20260620 01:53:04.594449 681 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260620 01:53:04.599766 699 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:04.602617 681 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:53:04.602932 681 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "b171c7ed0a74499abcaa971b8f349a97"
format_stamp: "Formatted at 2026-06-20 01:53:04 on dist-test-slave-wg14"
I20260620 01:53:04.603271 681 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:04.631503 681 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:04.632790 681 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:04.633141 681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:04.661023 681 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:36607
I20260620 01:53:04.660946 750 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:36607 every 8 connection(s)
I20260620 01:53:04.664379 681 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:53:04.666887 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 681
I20260620 01:53:04.667285 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:53:04.668910 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:33299
--webserver_interface=127.20.1.188
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299 with env {}
I20260620 01:53:04.674166 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:53:04.682936 751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:53:04.705304 751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:53:04.719676 751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:53:04.722594 700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33299: connect: Connection refused (error 111)
W20260620 01:53:04.725997 751 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33299: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33299: connect: Connection refused (error 111)
I20260620 01:53:04.735227 679 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } attempt: 5
I20260620 01:53:04.745227 679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:53:04.748273 679 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33299: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33299: connect: Connection refused (error 111)
I20260620 01:53:04.775457 751 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } attempt: 1
W20260620 01:53:04.778211 751 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33299: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33299: connect: Connection refused (error 111)
I20260620 01:53:04.799696 679 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } attempt: 1
W20260620 01:53:04.803556 679 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33299: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33299: connect: Connection refused (error 111)
I20260620 01:53:04.846542 751 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } attempt: 2
W20260620 01:53:04.849313 751 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33299: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33299: connect: Connection refused (error 111)
I20260620 01:53:04.871904 679 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } attempt: 2
W20260620 01:53:04.875522 679 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33299: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33299: connect: Connection refused (error 111)
I20260620 01:53:04.940665 751 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } attempt: 3
W20260620 01:53:04.943495 751 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33299: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33299: connect: Connection refused (error 111)
I20260620 01:53:04.981907 679 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } attempt: 3
W20260620 01:53:04.984817 679 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33299: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33299: connect: Connection refused (error 111)
I20260620 01:53:05.086823 751 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } attempt: 4
W20260620 01:53:05.089977 751 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33299: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33299: connect: Connection refused (error 111)
W20260620 01:53:05.119735 753 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:05.120175 753 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:05.120297 753 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:05.130257 753 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:05.130383 753 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:05.130441 753 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:05.130504 753 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:05.142941 753 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:33299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:05.145097 753 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:05.147712 753 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:05.158372 762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:05.158316 763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:05.159593 765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:05.160072 753 server_base.cc:1061] running on GCE node
I20260620 01:53:05.161207 753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:05.162179 679 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } attempt: 4
I20260620 01:53:05.163932 753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260620 01:53:05.165419 679 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33299: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33299: connect: Connection refused (error 111)
I20260620 01:53:05.167659 753 hybrid_clock.cc:648] HybridClock initialized: now 1781920385167385 us; error 129 us; skew 500 ppm
I20260620 01:53:05.168716 753 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:05.172214 753 webserver.cc:533] Webserver started at http://127.20.1.188:34313/ using document root <none> and password file <none>
I20260620 01:53:05.173401 753 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:05.173632 753 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:05.174118 753 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:53:05.177343 753 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "8781c9b655eb4aeca5d56a909dc1211e"
format_stamp: "Formatted at 2026-06-20 01:53:05 on dist-test-slave-wg14"
I20260620 01:53:05.178476 753 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "8781c9b655eb4aeca5d56a909dc1211e"
format_stamp: "Formatted at 2026-06-20 01:53:05 on dist-test-slave-wg14"
I20260620 01:53:05.186434 753 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260620 01:53:05.191890 771 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:05.193995 753 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:53:05.194234 753 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "8781c9b655eb4aeca5d56a909dc1211e"
format_stamp: "Formatted at 2026-06-20 01:53:05 on dist-test-slave-wg14"
I20260620 01:53:05.194540 753 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:05.226970 753 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:05.228219 753 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:05.228583 753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:05.256454 753 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:33299
I20260620 01:53:05.256464 822 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:33299 every 8 connection(s)
I20260620 01:53:05.258394 753 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:53:05.265137 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 753
I20260620 01:53:05.265291 823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:53:05.265411 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:53:05.277241 823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:53:05.301327 823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:53:05.310958 823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:53:05.328426 823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e: Bootstrap starting.
I20260620 01:53:05.332428 823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e: Neither blocks nor log segments found. Creating new log.
I20260620 01:53:05.334115 823 log.cc:826] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:05.338418 823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e: No bootstrap required, opened a new log
I20260620 01:53:05.344563 823 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:05.345791 823 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:53:05.346122 823 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8781c9b655eb4aeca5d56a909dc1211e, State: Initialized, Role: FOLLOWER
I20260620 01:53:05.347117 823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:05.348389 833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:05.348707 833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:05.349877 823 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:53:05.366994 844 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:53:05.367448 844 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:53:05.363533 753 master_runner.cc:187] Error getting master registration for 127.20.1.190:40263: OK, instance_id { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" instance_seqno: 1781920384013622 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:53:05.371579 753 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:53:05.389356 751 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } attempt: 5
I20260620 01:53:05.403784 751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97: Bootstrap starting.
I20260620 01:53:05.409279 751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97: Neither blocks nor log segments found. Creating new log.
I20260620 01:53:05.411170 751 log.cc:826] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:05.415558 751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97: No bootstrap required, opened a new log
I20260620 01:53:05.424086 751 raft_consensus.cc:359] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:05.424612 751 raft_consensus.cc:385] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:53:05.424767 751 raft_consensus.cc:740] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b171c7ed0a74499abcaa971b8f349a97, State: Initialized, Role: FOLLOWER
I20260620 01:53:05.425956 751 consensus_queue.cc:260] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:05.427475 847 sys_catalog.cc:455] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:05.427945 847 sys_catalog.cc:458] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:05.429072 751 sys_catalog.cc:565] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:53:05.442746 679 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } attempt: 5
I20260620 01:53:05.440891 681 master_runner.cc:187] Error getting master registration for 127.20.1.190:40263: OK, instance_id { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" instance_seqno: 1781920384013622 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:53:05.449059 681 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:53:05.454500 858 catalog_manager.cc:1593] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:53:05.454869 858 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:53:05.462929 679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: Bootstrap starting.
I20260620 01:53:05.467334 679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: Neither blocks nor log segments found. Creating new log.
I20260620 01:53:05.468931 679 log.cc:826] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:05.472918 679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: No bootstrap required, opened a new log
I20260620 01:53:05.473887 847 raft_consensus.cc:493] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:53:05.474185 847 raft_consensus.cc:515] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:05.481060 798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b171c7ed0a74499abcaa971b8f349a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8781c9b655eb4aeca5d56a909dc1211e" is_pre_election: true
I20260620 01:53:05.481947 798 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b171c7ed0a74499abcaa971b8f349a97 in term 0.
W20260620 01:53:05.482738 777 tablet.cc:2406] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:53:05.484722 700 leader_election.cc:304] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8781c9b655eb4aeca5d56a909dc1211e, b171c7ed0a74499abcaa971b8f349a97; no voters:
I20260620 01:53:05.480312 847 leader_election.cc:290] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 025e1c47b0614610ad9da9addf02bdaf (127.20.1.190:40263), 8781c9b655eb4aeca5d56a909dc1211e (127.20.1.188:33299)
I20260620 01:53:05.486275 679 raft_consensus.cc:359] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:05.486851 679 raft_consensus.cc:385] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:53:05.486508 847 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:53:05.487028 679 raft_consensus.cc:740] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025e1c47b0614610ad9da9addf02bdaf, State: Initialized, Role: FOLLOWER
I20260620 01:53:05.487658 847 raft_consensus.cc:493] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:53:05.488555 847 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:53:05.488369 679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:05.509022 862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:05.509617 862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:05.511246 679 sys_catalog.cc:565] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:53:05.511250 654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b171c7ed0a74499abcaa971b8f349a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025e1c47b0614610ad9da9addf02bdaf" is_pre_election: true
I20260620 01:53:05.512135 654 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b171c7ed0a74499abcaa971b8f349a97 in term 0.
I20260620 01:53:05.514762 847 raft_consensus.cc:515] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:05.519373 654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b171c7ed0a74499abcaa971b8f349a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025e1c47b0614610ad9da9addf02bdaf"
I20260620 01:53:05.519737 654 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:53:05.524350 654 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b171c7ed0a74499abcaa971b8f349a97 in term 1.
I20260620 01:53:05.526271 701 leader_election.cc:304] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025e1c47b0614610ad9da9addf02bdaf, b171c7ed0a74499abcaa971b8f349a97; no voters:
I20260620 01:53:05.527406 861 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:53:05.530390 798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b171c7ed0a74499abcaa971b8f349a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8781c9b655eb4aeca5d56a909dc1211e"
I20260620 01:53:05.530788 798 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:53:05.533121 847 leader_election.cc:290] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [CANDIDATE]: Term 1 election: Requested vote from peers 025e1c47b0614610ad9da9addf02bdaf (127.20.1.190:40263), 8781c9b655eb4aeca5d56a909dc1211e (127.20.1.188:33299)
I20260620 01:53:05.535365 798 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b171c7ed0a74499abcaa971b8f349a97 in term 1.
I20260620 01:53:05.541538 861 raft_consensus.cc:697] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 1 LEADER]: Becoming Leader. State: Replica: b171c7ed0a74499abcaa971b8f349a97, State: Running, Role: LEADER
I20260620 01:53:05.542640 861 consensus_queue.cc:237] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:05.558663 847 sys_catalog.cc:455] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b171c7ed0a74499abcaa971b8f349a97. Latest consensus state: current_term: 1 leader_uuid: "b171c7ed0a74499abcaa971b8f349a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:05.560204 847 sys_catalog.cc:458] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:05.571604 872 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:53:05.575893 872 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:53:05.593000 876 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 01:53:05.610363 798 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [term 1 FOLLOWER]: Refusing update from remote peer b171c7ed0a74499abcaa971b8f349a97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:53:05.612491 847 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:53:05.617110 654 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 1 FOLLOWER]: Refusing update from remote peer b171c7ed0a74499abcaa971b8f349a97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:53:05.619418 847 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 01:53:05.637321 705 tablet.cc:2406] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:53:05.642148 862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b171c7ed0a74499abcaa971b8f349a97. Latest consensus state: current_term: 1 leader_uuid: "b171c7ed0a74499abcaa971b8f349a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:05.642514 862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:05.653613 877 mvcc.cc:204] Tried to move back new op lower bound from 7298745899440230400 to 7298745899193647104. Current Snapshot: MvccSnapshot[applied={T|T < 7298745899440230400}]
I20260620 01:53:05.655512 847 sys_catalog.cc:455] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b171c7ed0a74499abcaa971b8f349a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:05.655840 847 sys_catalog.cc:458] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:05.658001 872 catalog_manager.cc:1382] Generated new cluster ID: 537bf1ccc4844dc3a6fa159d662de766
I20260620 01:53:05.658174 872 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:53:05.658527 879 mvcc.cc:204] Tried to move back new op lower bound from 7298745899440230400 to 7298745899193647104. Current Snapshot: MvccSnapshot[applied={T|T < 7298745899440230400}]
I20260620 01:53:05.660117 833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b171c7ed0a74499abcaa971b8f349a97. Latest consensus state: current_term: 1 leader_uuid: "b171c7ed0a74499abcaa971b8f349a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:05.660573 833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:05.669183 862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b171c7ed0a74499abcaa971b8f349a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:05.669517 862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:05.671736 866 sys_catalog.cc:455] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b171c7ed0a74499abcaa971b8f349a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:05.672164 866 sys_catalog.cc:458] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:05.690138 872 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:53:05.692840 872 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:53:05.701421 833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b171c7ed0a74499abcaa971b8f349a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:05.701769 833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:05.724145 872 catalog_manager.cc:6069] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97: Generated new TSK 0
I20260620 01:53:05.725808 872 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:53:05.758364 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260620 01:53:06.269896 885 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:06.270398 885 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:06.270535 885 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:06.286993 885 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:06.287758 885 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:53:06.302008 885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:06.303951 885 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:06.306282 885 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:06.321416 890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:06.321378 891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:06.321472 893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:06.322785 885 server_base.cc:1061] running on GCE node
I20260620 01:53:06.323652 885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:06.325915 885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:06.329600 885 hybrid_clock.cc:648] HybridClock initialized: now 1781920386329338 us; error 180 us; skew 500 ppm
I20260620 01:53:06.330150 885 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:06.333348 885 webserver.cc:533] Webserver started at http://127.20.1.129:35799/ using document root <none> and password file <none>
I20260620 01:53:06.334538 885 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:06.334785 885 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:06.335530 885 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:53:06.339432 885 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "923994cacde94cb1ba5837858b51375a"
format_stamp: "Formatted at 2026-06-20 01:53:06 on dist-test-slave-wg14"
I20260620 01:53:06.340994 885 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "923994cacde94cb1ba5837858b51375a"
format_stamp: "Formatted at 2026-06-20 01:53:06 on dist-test-slave-wg14"
I20260620 01:53:06.349329 885 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260620 01:53:06.356037 899 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:06.358585 885 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s
I20260620 01:53:06.358973 885 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "923994cacde94cb1ba5837858b51375a"
format_stamp: "Formatted at 2026-06-20 01:53:06 on dist-test-slave-wg14"
I20260620 01:53:06.359368 885 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:06.370474 844 catalog_manager.cc:1294] Loaded cluster ID: 537bf1ccc4844dc3a6fa159d662de766
I20260620 01:53:06.370601 844 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e: loading cluster ID for follower catalog manager: success
I20260620 01:53:06.376046 844 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e: acquiring CA information for follower catalog manager: success
I20260620 01:53:06.379695 844 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:53:06.390868 885 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:06.392055 885 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:06.392402 885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:06.394230 885 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:06.397217 885 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:06.397396 885 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:06.397567 885 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:06.397651 885 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:06.443825 885 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35309
I20260620 01:53:06.443909 1012 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35309 every 8 connection(s)
I20260620 01:53:06.445609 885 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:53:06.452994 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 885
I20260620 01:53:06.453202 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:53:06.461711 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:53:06.472440 1013 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33299
I20260620 01:53:06.472524 1015 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36607
I20260620 01:53:06.472538 1014 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40263
I20260620 01:53:06.473003 1013 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:06.473011 1014 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:06.473011 1015 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:06.474730 1013 heartbeater.cc:507] Master 127.20.1.188:33299 requested a full tablet report, sending...
I20260620 01:53:06.474763 1014 heartbeater.cc:507] Master 127.20.1.190:40263 requested a full tablet report, sending...
I20260620 01:53:06.474809 1015 heartbeater.cc:507] Master 127.20.1.189:36607 requested a full tablet report, sending...
I20260620 01:53:06.478605 644 ts_manager.cc:194] Registered new tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:35309)
I20260620 01:53:06.478615 716 ts_manager.cc:194] Registered new tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:35309)
I20260620 01:53:06.478637 788 ts_manager.cc:194] Registered new tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:35309)
I20260620 01:53:06.481302 716 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:38851
I20260620 01:53:06.599427 876 catalog_manager.cc:1294] Loaded cluster ID: 537bf1ccc4844dc3a6fa159d662de766
I20260620 01:53:06.599622 876 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: loading cluster ID for follower catalog manager: success
I20260620 01:53:06.605679 876 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: acquiring CA information for follower catalog manager: success
I20260620 01:53:06.610682 876 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 01:53:06.910650 1023 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:06.911213 1023 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:06.911424 1023 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:06.922792 1023 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:06.923025 1023 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:53:06.935067 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:06.937191 1023 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:06.940032 1023 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:06.954514 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:06.955626 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:06.957479 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:06.957859 1023 server_base.cc:1061] running on GCE node
I20260620 01:53:06.958628 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:06.960215 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:06.963807 1023 hybrid_clock.cc:648] HybridClock initialized: now 1781920386963533 us; error 185 us; skew 500 ppm
I20260620 01:53:06.964359 1023 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:06.968911 1023 webserver.cc:533] Webserver started at http://127.20.1.130:35027/ using document root <none> and password file <none>
I20260620 01:53:06.970278 1023 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:06.970558 1023 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:06.971005 1023 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:53:06.974503 1023 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/data/instance:
uuid: "f7e99ead91bc4b29a7a4675f29b8f13c"
format_stamp: "Formatted at 2026-06-20 01:53:06 on dist-test-slave-wg14"
I20260620 01:53:06.975639 1023 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance:
uuid: "f7e99ead91bc4b29a7a4675f29b8f13c"
format_stamp: "Formatted at 2026-06-20 01:53:06 on dist-test-slave-wg14"
I20260620 01:53:06.984197 1023 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s
I20260620 01:53:06.989460 1038 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:06.992533 1023 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:53:06.992836 1023 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "f7e99ead91bc4b29a7a4675f29b8f13c"
format_stamp: "Formatted at 2026-06-20 01:53:06 on dist-test-slave-wg14"
I20260620 01:53:06.993145 1023 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:07.016242 1023 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:07.017338 1023 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:07.017678 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:07.018929 1023 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:07.021351 1023 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:07.021503 1023 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:07.021617 1023 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:07.021703 1023 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:07.076597 1023 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:45739
I20260620 01:53:07.076701 1151 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:45739 every 8 connection(s)
I20260620 01:53:07.078329 1023 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:53:07.081241 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 1023
I20260620 01:53:07.081528 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
I20260620 01:53:07.087015 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:53:07.128403 1154 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36607
I20260620 01:53:07.128461 1153 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40263
I20260620 01:53:07.129122 1153 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:07.129169 1154 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:07.130280 1152 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33299
I20260620 01:53:07.130494 1152 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:07.131081 1154 heartbeater.cc:507] Master 127.20.1.189:36607 requested a full tablet report, sending...
I20260620 01:53:07.131119 1153 heartbeater.cc:507] Master 127.20.1.190:40263 requested a full tablet report, sending...
I20260620 01:53:07.131392 1152 heartbeater.cc:507] Master 127.20.1.188:33299 requested a full tablet report, sending...
I20260620 01:53:07.133360 716 ts_manager.cc:194] Registered new tserver with Master: f7e99ead91bc4b29a7a4675f29b8f13c (127.20.1.130:45739)
I20260620 01:53:07.133437 788 ts_manager.cc:194] Registered new tserver with Master: f7e99ead91bc4b29a7a4675f29b8f13c (127.20.1.130:45739)
I20260620 01:53:07.133481 644 ts_manager.cc:194] Registered new tserver with Master: f7e99ead91bc4b29a7a4675f29b8f13c (127.20.1.130:45739)
I20260620 01:53:07.134582 716 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.130:59199
I20260620 01:53:07.486061 1015 heartbeater.cc:499] Master 127.20.1.189:36607 was elected leader, sending a full tablet report...
W20260620 01:53:07.530637 1162 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:07.531035 1162 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:07.531180 1162 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:07.542404 1162 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:07.542620 1162 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:53:07.557126 1162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:07.559350 1162 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:07.561820 1162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:07.578186 1167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:07.578200 1168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:07.578398 1170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:07.581357 1162 server_base.cc:1061] running on GCE node
I20260620 01:53:07.582208 1162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:07.584102 1162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:07.587735 1162 hybrid_clock.cc:648] HybridClock initialized: now 1781920387587479 us; error 172 us; skew 500 ppm
I20260620 01:53:07.588258 1162 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:07.591439 1162 webserver.cc:533] Webserver started at http://127.20.1.131:37739/ using document root <none> and password file <none>
I20260620 01:53:07.592423 1162 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:07.592573 1162 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:07.592978 1162 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:53:07.596305 1162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/data/instance:
uuid: "e654c72cf9a54ddb802846ce8cd48074"
format_stamp: "Formatted at 2026-06-20 01:53:07 on dist-test-slave-wg14"
I20260620 01:53:07.597473 1162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance:
uuid: "e654c72cf9a54ddb802846ce8cd48074"
format_stamp: "Formatted at 2026-06-20 01:53:07 on dist-test-slave-wg14"
I20260620 01:53:07.605757 1162 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.001s sys 0.007s
I20260620 01:53:07.612421 1176 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:07.614945 1162 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:53:07.615243 1162 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "e654c72cf9a54ddb802846ce8cd48074"
format_stamp: "Formatted at 2026-06-20 01:53:07 on dist-test-slave-wg14"
I20260620 01:53:07.615556 1162 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:07.644243 1162 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:07.645756 1162 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:07.646152 1162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:07.647444 1162 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:07.649762 1162 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:07.649899 1162 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:07.649999 1162 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:07.650058 1162 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:07.700649 1162 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:37631
I20260620 01:53:07.700706 1289 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:37631 every 8 connection(s)
I20260620 01:53:07.702471 1162 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:53:07.711287 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 1162
I20260620 01:53:07.711540 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
I20260620 01:53:07.745467 1292 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36607
I20260620 01:53:07.745805 1290 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33299
I20260620 01:53:07.746119 1292 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:07.746119 1290 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:07.748104 1290 heartbeater.cc:507] Master 127.20.1.188:33299 requested a full tablet report, sending...
I20260620 01:53:07.748253 1291 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40263
I20260620 01:53:07.748618 1291 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:07.748106 1292 heartbeater.cc:507] Master 127.20.1.189:36607 requested a full tablet report, sending...
I20260620 01:53:07.749342 1291 heartbeater.cc:507] Master 127.20.1.190:40263 requested a full tablet report, sending...
I20260620 01:53:07.751318 788 ts_manager.cc:194] Registered new tserver with Master: e654c72cf9a54ddb802846ce8cd48074 (127.20.1.131:37631)
I20260620 01:53:07.751318 644 ts_manager.cc:194] Registered new tserver with Master: e654c72cf9a54ddb802846ce8cd48074 (127.20.1.131:37631)
I20260620 01:53:07.751682 716 ts_manager.cc:194] Registered new tserver with Master: e654c72cf9a54ddb802846ce8cd48074 (127.20.1.131:37631)
I20260620 01:53:07.753120 716 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.131:60145
I20260620 01:53:07.758739 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:53:07.776304 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:53:07.776551 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 613
I20260620 01:53:07.777885 674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:07.914551 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 613
I20260620 01:53:07.939677 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40263
--webserver_interface=127.20.1.190
--webserver_port=36149
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299 with env {}
I20260620 01:53:08.138134 1154 heartbeater.cc:499] Master 127.20.1.189:36607 was elected leader, sending a full tablet report...
W20260620 01:53:08.138415 1153 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:40263 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:40263: connect: Connection refused (error 111)
W20260620 01:53:08.178426 701 consensus_peers.cc:597] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 -> Peer 025e1c47b0614610ad9da9addf02bdaf (127.20.1.190:40263): Couldn't send request to peer 025e1c47b0614610ad9da9addf02bdaf. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:40263: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260620 01:53:08.422375 1302 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:08.422843 1302 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:08.422982 1302 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:08.433084 1302 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:08.433238 1302 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:08.433323 1302 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:08.433393 1302 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:08.445798 1302 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=36149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:08.448225 1302 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:08.450647 1302 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:08.462659 1310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:08.462988 1313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:08.463258 1311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:08.464447 1302 server_base.cc:1061] running on GCE node
I20260620 01:53:08.465647 1302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:08.468345 1302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:08.471907 1302 hybrid_clock.cc:648] HybridClock initialized: now 1781920388471703 us; error 110 us; skew 500 ppm
I20260620 01:53:08.472455 1302 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:08.476051 1302 webserver.cc:533] Webserver started at http://127.20.1.190:36149/ using document root <none> and password file <none>
I20260620 01:53:08.477015 1302 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:08.477198 1302 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260620 01:53:08.485895 1014 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:40263 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:40263: connect: Connection refused (error 111)
I20260620 01:53:08.485877 1302 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.003s
I20260620 01:53:08.490509 1320 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:08.492800 1302 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:53:08.493047 1302 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "025e1c47b0614610ad9da9addf02bdaf"
format_stamp: "Formatted at 2026-06-20 01:53:03 on dist-test-slave-wg14"
I20260620 01:53:08.494642 1302 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:08.516903 1302 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:08.518996 1302 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:08.519596 1302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:08.549971 1302 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:40263
I20260620 01:53:08.550026 1371 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:40263 every 8 connection(s)
I20260620 01:53:08.552430 1302 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:53:08.553184 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 1302
I20260620 01:53:08.553574 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 885
I20260620 01:53:08.564946 1372 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:53:08.569103 1008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:08.570582 1372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: Bootstrap starting.
I20260620 01:53:08.592374 1372 log.cc:826] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:08.606330 1372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:53:08.607177 1372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: Bootstrap complete.
I20260620 01:53:08.617590 1372 raft_consensus.cc:359] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:08.618503 1372 raft_consensus.cc:740] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025e1c47b0614610ad9da9addf02bdaf, State: Initialized, Role: FOLLOWER
I20260620 01:53:08.619702 1372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:08.620980 1378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:08.621412 1378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:08.623179 1372 sys_catalog.cc:565] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:53:08.637126 1389 catalog_manager.cc:1294] Loaded cluster ID: 537bf1ccc4844dc3a6fa159d662de766
I20260620 01:53:08.637455 1389 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: loading cluster ID for follower catalog manager: success
I20260620 01:53:08.644443 1389 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: acquiring CA information for follower catalog manager: success
I20260620 01:53:08.652346 1389 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:53:08.705775 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 885
I20260620 01:53:08.731637 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35309
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--webserver_port=35799
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:53:08.734786 861 sys_catalog.cc:455] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b171c7ed0a74499abcaa971b8f349a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:08.735199 861 sys_catalog.cc:458] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:08.752297 1378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b171c7ed0a74499abcaa971b8f349a97. Latest consensus state: current_term: 1 leader_uuid: "b171c7ed0a74499abcaa971b8f349a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:08.753084 1378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:08.756992 1292 heartbeater.cc:499] Master 127.20.1.189:36607 was elected leader, sending a full tablet report...
I20260620 01:53:08.768975 1337 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e654c72cf9a54ddb802846ce8cd48074" instance_seqno: 1781920387686936) as {username='slave'} at 127.20.1.131:52185; Asking this server to re-register.
I20260620 01:53:08.770414 1291 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:08.773063 1337 ts_manager.cc:194] Registered new tserver with Master: e654c72cf9a54ddb802846ce8cd48074 (127.20.1.131:37631)
I20260620 01:53:09.149012 1153 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40263
I20260620 01:53:09.150920 1337 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f7e99ead91bc4b29a7a4675f29b8f13c" instance_seqno: 1781920387063235) as {username='slave'} at 127.20.1.130:56619; Asking this server to re-register.
I20260620 01:53:09.152120 1153 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:09.155579 1337 ts_manager.cc:194] Registered new tserver with Master: f7e99ead91bc4b29a7a4675f29b8f13c (127.20.1.130:45739)
W20260620 01:53:09.220180 1399 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:09.220568 1399 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:09.220670 1399 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:09.232223 1399 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:09.232484 1399 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:53:09.245218 1399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35309
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=35799
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:09.247105 1399 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:09.249609 1399 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:09.264050 1407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:09.266819 1408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:09.268319 1410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:09.268899 1399 server_base.cc:1061] running on GCE node
I20260620 01:53:09.269773 1399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:09.271487 1399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:09.275058 1399 hybrid_clock.cc:648] HybridClock initialized: now 1781920389274697 us; error 267 us; skew 500 ppm
I20260620 01:53:09.275700 1399 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:09.278894 1399 webserver.cc:533] Webserver started at http://127.20.1.129:35799/ using document root <none> and password file <none>
I20260620 01:53:09.280125 1396 sys_catalog.cc:455] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b171c7ed0a74499abcaa971b8f349a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:09.280423 1396 sys_catalog.cc:458] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:09.280602 1399 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:09.280839 1399 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:09.292882 1399 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 01:53:09.298193 1419 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:09.300666 1399 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.004s
I20260620 01:53:09.300969 1399 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "923994cacde94cb1ba5837858b51375a"
format_stamp: "Formatted at 2026-06-20 01:53:06 on dist-test-slave-wg14"
I20260620 01:53:09.302258 1399 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:09.332605 1399 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:09.333774 1399 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:09.334137 1399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:09.335676 1399 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:09.338198 1399 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:09.338352 1399 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:09.338511 1399 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:09.338572 1399 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:09.382109 1399 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35309
I20260620 01:53:09.382174 1532 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35309 every 8 connection(s)
I20260620 01:53:09.384079 1399 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:53:09.390892 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 1399
I20260620 01:53:09.410791 1534 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40263
I20260620 01:53:09.411355 1533 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33299
I20260620 01:53:09.411620 1533 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:09.411639 1534 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:09.413987 1533 heartbeater.cc:507] Master 127.20.1.188:33299 requested a full tablet report, sending...
I20260620 01:53:09.413987 1534 heartbeater.cc:507] Master 127.20.1.190:40263 requested a full tablet report, sending...
I20260620 01:53:09.416857 788 ts_manager.cc:194] Re-registered known tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:35309)
I20260620 01:53:09.417045 1337 ts_manager.cc:194] Registered new tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:35309)
I20260620 01:53:09.420877 1535 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36607
I20260620 01:53:09.421134 1535 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:09.421672 1535 heartbeater.cc:507] Master 127.20.1.189:36607 requested a full tablet report, sending...
I20260620 01:53:09.423098 716 ts_manager.cc:194] Re-registered known tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:35309)
I20260620 01:53:09.424072 716 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:55309
I20260620 01:53:09.457163 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 1399
I20260620 01:53:09.479236 1528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:09.593973 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 1399
I20260620 01:53:09.611994 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35309
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--webserver_port=35799
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
W20260620 01:53:10.091575 1547 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:10.091943 1547 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:10.092041 1547 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:53:10.092101 1547 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:10.102087 1547 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:10.102344 1547 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:53:10.115082 1547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35309
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=35799
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:10.117199 1547 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:10.119768 1547 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:10.135715 1552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:10.136055 1553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:10.136029 1555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:10.139674 1547 server_base.cc:1061] running on GCE node
I20260620 01:53:10.141808 1547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:10.144307 1547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:10.148186 1547 hybrid_clock.cc:648] HybridClock initialized: now 1781920390147694 us; error 377 us; skew 500 ppm
I20260620 01:53:10.148650 1547 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:10.152607 1547 webserver.cc:533] Webserver started at http://127.20.1.129:35799/ using document root <none> and password file <none>
I20260620 01:53:10.153648 1547 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:10.153838 1547 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:10.162039 1547 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:53:10.167261 1561 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:10.170763 1547 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260620 01:53:10.171087 1547 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "923994cacde94cb1ba5837858b51375a"
format_stamp: "Formatted at 2026-06-20 01:53:06 on dist-test-slave-wg14"
I20260620 01:53:10.172732 1547 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:10.199352 1547 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:10.201121 1547 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:10.201567 1547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:10.203541 1547 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:10.206629 1547 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:10.206835 1547 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:10.206980 1547 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:10.207077 1547 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:53:10.207645 1547 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:53:10.207765 1547 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:53:10.207854 1547 fault_injection.cc:59] injecting fault for kudu (pid 1547): FLAGS_fault_before_start (process will exit)
I20260620 01:53:10.224784 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35309
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--webserver_port=35799
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260620 01:53:10.698222 1570 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:10.698611 1570 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:10.698736 1570 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:10.710073 1570 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:10.710342 1570 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:53:10.724187 1570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45091,127.20.1.149:38399,127.20.1.150:42563,127.20.1.151:33329,127.20.1.152:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35309
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=35799
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:10.726845 1570 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:10.729430 1570 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:10.743844 1577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:10.744216 1578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:10.744841 1580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:10.747066 1570 server_base.cc:1061] running on GCE node
I20260620 01:53:10.748306 1570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:10.750013 1570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:10.753602 1570 hybrid_clock.cc:648] HybridClock initialized: now 1781920390753335 us; error 160 us; skew 500 ppm
I20260620 01:53:10.754209 1570 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:10.758011 1570 webserver.cc:533] Webserver started at http://127.20.1.129:35799/ using document root <none> and password file <none>
I20260620 01:53:10.759613 1570 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:10.760116 1570 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:10.769062 1570 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 01:53:10.773340 1586 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:10.775472 1570 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:53:10.775719 1570 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "923994cacde94cb1ba5837858b51375a"
format_stamp: "Formatted at 2026-06-20 01:53:06 on dist-test-slave-wg14"
I20260620 01:53:10.776723 1570 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:10.799238 1570 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:10.800505 1570 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:10.800879 1570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:10.802796 1570 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:10.805801 1570 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:10.805971 1570 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:10.806143 1570 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:10.806206 1570 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:10.850878 1570 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35309
I20260620 01:53:10.850936 1699 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35309 every 8 connection(s)
I20260620 01:53:10.853004 1570 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:53:10.856150 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 1570
I20260620 01:53:10.856520 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 1570
I20260620 01:53:10.893296 1703 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36607
I20260620 01:53:10.893425 1702 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40263
I20260620 01:53:10.893940 1702 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:10.893949 1703 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:10.893235 1701 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33299
I20260620 01:53:10.894312 1701 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:10.895337 1695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:11.000178 1702 heartbeater.cc:507] Master 127.20.1.190:40263 requested a full tablet report, sending...
I20260620 01:53:11.000224 1701 heartbeater.cc:507] Master 127.20.1.188:33299 requested a full tablet report, sending...
I20260620 01:53:11.001098 1703 heartbeater.cc:507] Master 127.20.1.189:36607 requested a full tablet report, sending...
I20260620 01:53:11.002954 788 ts_manager.cc:194] Re-registered known tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:35309)
I20260620 01:53:11.002974 716 ts_manager.cc:194] Re-registered known tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:35309)
I20260620 01:53:11.003523 1337 ts_manager.cc:194] Re-registered known tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:35309)
I20260620 01:53:11.004333 716 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:60057
I20260620 01:53:11.005487 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 1570
I20260620 01:53:11.021518 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 1023
W20260620 01:53:11.021546 775 connection.cc:573] server connection from 127.20.1.129:46467 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:53:11.034636 1147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:11.143041 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 1023
I20260620 01:53:11.161096 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 1162
I20260620 01:53:11.173070 1285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:11.284802 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 1162
I20260620 01:53:11.300133 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 1302
I20260620 01:53:11.309747 1367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:11.444638 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 1302
I20260620 01:53:11.464111 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 681
I20260620 01:53:11.465837 746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:11.597448 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 681
I20260620 01:53:11.619241 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 753
I20260620 01:53:11.620524 818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:11.748941 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 753
Shutting down hive metastore.
2026-06-20T01:53:12Z chronyd exiting
2026-06-20T01:53:12Z chronyd exiting
2026-06-20T01:53:12Z chronyd exiting
2026-06-20T01:53:12Z chronyd exiting
2026-06-20T01:53:12Z chronyd exiting
2026-06-20T01:53:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:53:12Z Disabled control of system clock
2026-06-20T01:53:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:53:12Z Disabled control of system clock
2026-06-20T01:53:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:53:12Z Disabled control of system clock
2026-06-20T01:53:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:53:12Z Disabled control of system clock
2026-06-20T01:53:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:53:12Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/security.properties
2026-06-20 01:53:13: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260620 01:53:24.176271 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40263
--webserver_interface=127.20.1.190
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299 with env {}
W20260620 01:53:24.692756 1915 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:24.693290 1915 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:24.693445 1915 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:24.705267 1915 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:24.705439 1915 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:24.705526 1915 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:24.705595 1915 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:24.718741 1915 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:24.720947 1915 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:24.723402 1915 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:24.736534 1921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:24.736585 1920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:24.736727 1915 server_base.cc:1061] running on GCE node
W20260620 01:53:24.736899 1923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:24.738160 1915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:24.740701 1915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:24.744233 1915 hybrid_clock.cc:648] HybridClock initialized: now 1781920404744006 us; error 130 us; skew 500 ppm
I20260620 01:53:24.744959 1915 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:24.749131 1915 webserver.cc:533] Webserver started at http://127.20.1.190:39429/ using document root <none> and password file <none>
I20260620 01:53:24.750532 1915 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:24.750715 1915 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:24.760192 1915 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:53:24.766000 1929 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:24.768172 1915 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:53:24.768446 1915 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "025e1c47b0614610ad9da9addf02bdaf"
format_stamp: "Formatted at 2026-06-20 01:53:03 on dist-test-slave-wg14"
I20260620 01:53:24.769647 1915 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:24.793231 1915 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:24.794458 1915 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:24.794822 1915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:24.822603 1915 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:40263
I20260620 01:53:24.822609 1980 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:40263 every 8 connection(s)
I20260620 01:53:24.824956 1915 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:53:24.834249 1981 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:53:24.834769 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 1915
I20260620 01:53:24.835004 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:53:24.836578 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:36607
--webserver_interface=127.20.1.189
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299 with env {}
I20260620 01:53:24.839814 1981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: Bootstrap starting.
I20260620 01:53:24.865921 1981 log.cc:826] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:24.883618 1981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:53:24.884505 1981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: Bootstrap complete.
I20260620 01:53:24.895625 1981 raft_consensus.cc:359] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:24.896821 1981 raft_consensus.cc:740] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025e1c47b0614610ad9da9addf02bdaf, State: Initialized, Role: FOLLOWER
I20260620 01:53:24.897979 1981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:24.899065 1985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:24.899431 1985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:24.901119 1981 sys_catalog.cc:565] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:53:24.915475 1996 catalog_manager.cc:1294] Loaded cluster ID: 537bf1ccc4844dc3a6fa159d662de766
I20260620 01:53:24.915807 1996 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: loading cluster ID for follower catalog manager: success
I20260620 01:53:24.923732 1996 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: acquiring CA information for follower catalog manager: success
I20260620 01:53:24.929724 1996 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:53:24.947474 1915 master_runner.cc:187] Error getting master registration for 127.20.1.189:36607: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36607: connect: Connection refused (error 111),
I20260620 01:53:24.953824 1915 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:53:25.351330 1983 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:25.351706 1983 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:25.351868 1983 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:25.361917 1983 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:25.362102 1983 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:25.362214 1983 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:25.362300 1983 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:25.376957 1983 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:36607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:25.378918 1983 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:25.381493 1983 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:25.392292 2004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:25.393025 2005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:25.395268 2007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:25.395642 1983 server_base.cc:1061] running on GCE node
I20260620 01:53:25.397027 1983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:25.400403 1983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:25.403986 1983 hybrid_clock.cc:648] HybridClock initialized: now 1781920405403712 us; error 161 us; skew 500 ppm
I20260620 01:53:25.404665 1983 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:25.409241 1983 webserver.cc:533] Webserver started at http://127.20.1.189:33309/ using document root <none> and password file <none>
I20260620 01:53:25.410528 1983 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:25.410687 1983 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:25.419658 1983 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:53:25.424510 2013 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:25.427250 1983 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:53:25.427536 1983 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "b171c7ed0a74499abcaa971b8f349a97"
format_stamp: "Formatted at 2026-06-20 01:53:04 on dist-test-slave-wg14"
I20260620 01:53:25.428521 1983 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:25.446566 1983 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:25.447742 1983 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:25.448035 1983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:25.476210 1983 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:36607
I20260620 01:53:25.476248 2065 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:36607 every 8 connection(s)
I20260620 01:53:25.478701 1983 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:53:25.482064 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 1983
I20260620 01:53:25.482308 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:53:25.483669 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:33299
--webserver_interface=127.20.1.188
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299 with env {}
I20260620 01:53:25.490300 2066 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:53:25.494925 2066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97: Bootstrap starting.
I20260620 01:53:25.516917 2066 log.cc:826] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:25.529521 2066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:53:25.530174 2066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97: Bootstrap complete.
I20260620 01:53:25.538086 2066 raft_consensus.cc:359] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:25.538937 2066 raft_consensus.cc:740] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b171c7ed0a74499abcaa971b8f349a97, State: Initialized, Role: FOLLOWER
I20260620 01:53:25.539990 2066 consensus_queue.cc:260] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:25.540999 2070 sys_catalog.cc:455] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:25.541332 2070 sys_catalog.cc:458] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:25.542311 2066 sys_catalog.cc:565] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:53:25.571810 2081 catalog_manager.cc:1294] Loaded cluster ID: 537bf1ccc4844dc3a6fa159d662de766
I20260620 01:53:25.572011 2081 catalog_manager.cc:1587] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97: loading cluster ID for follower catalog manager: success
I20260620 01:53:25.579392 2081 catalog_manager.cc:1609] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97: acquiring CA information for follower catalog manager: success
I20260620 01:53:25.583184 2081 catalog_manager.cc:1637] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:53:25.613207 1983 master_runner.cc:187] Error getting master registration for 127.20.1.188:33299: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33299: connect: Connection refused (error 111),
I20260620 01:53:25.620198 1983 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:53:25.965508 2068 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:25.965900 2068 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:25.966007 2068 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:25.977324 2068 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:25.977499 2068 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:25.977576 2068 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:25.977643 2068 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:25.993999 2068 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:33299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:25.996189 2068 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:25.998670 2068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:26.012070 2090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:26.014453 2093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:26.014634 2091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:26.017916 2068 server_base.cc:1061] running on GCE node
I20260620 01:53:26.019304 2068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:26.022212 2068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:26.025923 2068 hybrid_clock.cc:648] HybridClock initialized: now 1781920406025637 us; error 108 us; skew 500 ppm
I20260620 01:53:26.026656 2068 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:26.032995 2068 webserver.cc:533] Webserver started at http://127.20.1.188:39721/ using document root <none> and password file <none>
I20260620 01:53:26.035070 2068 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:26.035465 2068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:26.044432 2068 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 01:53:26.050179 2099 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:26.052487 2068 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
I20260620 01:53:26.052757 2068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "8781c9b655eb4aeca5d56a909dc1211e"
format_stamp: "Formatted at 2026-06-20 01:53:05 on dist-test-slave-wg14"
I20260620 01:53:26.053659 2068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:26.078644 2068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:26.080245 2068 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:26.080614 2068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:26.106340 2068 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:33299
I20260620 01:53:26.106343 2150 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:33299 every 8 connection(s)
I20260620 01:53:26.108810 2068 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:53:26.116588 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 2068
I20260620 01:53:26.116842 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:53:26.122277 2151 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:53:26.128791 2151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e: Bootstrap starting.
I20260620 01:53:26.146119 2157 raft_consensus.cc:493] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:53:26.146553 2157 raft_consensus.cc:515] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:26.155506 2157 leader_election.cc:290] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b171c7ed0a74499abcaa971b8f349a97 (127.20.1.189:36607), 8781c9b655eb4aeca5d56a909dc1211e (127.20.1.188:33299)
I20260620 01:53:26.166204 2151 log.cc:826] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:26.178519 2041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "025e1c47b0614610ad9da9addf02bdaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b171c7ed0a74499abcaa971b8f349a97" is_pre_election: true
I20260620 01:53:26.179229 2041 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 025e1c47b0614610ad9da9addf02bdaf in term 1.
I20260620 01:53:26.180032 1933 leader_election.cc:304] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025e1c47b0614610ad9da9addf02bdaf, b171c7ed0a74499abcaa971b8f349a97; no voters:
I20260620 01:53:26.180650 2157 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:53:26.180821 2157 raft_consensus.cc:493] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:53:26.180938 2157 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:53:26.187011 2157 raft_consensus.cc:515] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:26.188781 2151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:53:26.189229 2157 leader_election.cc:290] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [CANDIDATE]: Term 2 election: Requested vote from peers b171c7ed0a74499abcaa971b8f349a97 (127.20.1.189:36607), 8781c9b655eb4aeca5d56a909dc1211e (127.20.1.188:33299)
I20260620 01:53:26.189416 2041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "025e1c47b0614610ad9da9addf02bdaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b171c7ed0a74499abcaa971b8f349a97"
I20260620 01:53:26.189700 2041 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:53:26.189832 2151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e: Bootstrap complete.
I20260620 01:53:26.196911 2041 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025e1c47b0614610ad9da9addf02bdaf in term 2.
I20260620 01:53:26.197708 1933 leader_election.cc:304] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025e1c47b0614610ad9da9addf02bdaf, b171c7ed0a74499abcaa971b8f349a97; no voters:
I20260620 01:53:26.198284 2157 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 2 FOLLOWER]: Leader election won for term 2
I20260620 01:53:26.199116 2157 raft_consensus.cc:697] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 2 LEADER]: Becoming Leader. State: Replica: 025e1c47b0614610ad9da9addf02bdaf, State: Running, Role: LEADER
I20260620 01:53:26.200163 2157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:26.200131 2151 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:26.201288 2151 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8781c9b655eb4aeca5d56a909dc1211e, State: Initialized, Role: FOLLOWER
I20260620 01:53:26.202315 2151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:26.203610 2162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 025e1c47b0614610ad9da9addf02bdaf. Latest consensus state: current_term: 2 leader_uuid: "025e1c47b0614610ad9da9addf02bdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:26.204047 2162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:26.212407 2164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:26.213184 2164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:26.214093 2151 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:53:26.214093 2125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "025e1c47b0614610ad9da9addf02bdaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8781c9b655eb4aeca5d56a909dc1211e"
I20260620 01:53:26.214228 2126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "025e1c47b0614610ad9da9addf02bdaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8781c9b655eb4aeca5d56a909dc1211e" is_pre_election: true
I20260620 01:53:26.214552 2165 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:53:26.214905 2125 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:53:26.218909 2165 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:53:26.220314 2165 catalog_manager.cc:1294] Loaded cluster ID: 537bf1ccc4844dc3a6fa159d662de766
I20260620 01:53:26.220460 2165 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:53:26.221886 2165 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:53:26.223062 2165 catalog_manager.cc:6080] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: Loaded TSK: 0
I20260620 01:53:26.224601 2165 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:53:26.224496 2125 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025e1c47b0614610ad9da9addf02bdaf in term 2.
I20260620 01:53:26.233764 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:53:26.246137 2176 catalog_manager.cc:1294] Loaded cluster ID: 537bf1ccc4844dc3a6fa159d662de766
I20260620 01:53:26.246296 2176 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e: loading cluster ID for follower catalog manager: success
I20260620 01:53:26.252028 2176 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e: acquiring CA information for follower catalog manager: success
I20260620 01:53:26.256156 2176 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:53:26.587368 2041 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 2 FOLLOWER]: Refusing update from remote peer 025e1c47b0614610ad9da9addf02bdaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:53:26.589038 2162 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:53:26.596791 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 025e1c47b0614610ad9da9addf02bdaf. Latest consensus state: current_term: 2 leader_uuid: "025e1c47b0614610ad9da9addf02bdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:26.597275 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:26.609845 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "025e1c47b0614610ad9da9addf02bdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:26.610183 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:26.613817 2157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "025e1c47b0614610ad9da9addf02bdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:26.614122 2125 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [term 2 FOLLOWER]: Refusing update from remote peer 025e1c47b0614610ad9da9addf02bdaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:53:26.614379 2157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:26.615077 2157 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:53:26.625264 2188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "025e1c47b0614610ad9da9addf02bdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:26.625617 2164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 025e1c47b0614610ad9da9addf02bdaf. Latest consensus state: current_term: 2 leader_uuid: "025e1c47b0614610ad9da9addf02bdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:26.625957 2164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:26.626220 2188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:53:26.702984 2180 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:26.703480 2180 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:26.703641 2180 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:26.715355 2180 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:26.715629 2180 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:53:26.729884 2180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:26.732156 2180 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:26.734380 2180 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:26.749512 2196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:26.749429 2193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:26.749682 2194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:26.750985 2180 server_base.cc:1061] running on GCE node
I20260620 01:53:26.752547 2180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:26.754458 2180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:26.757932 2180 hybrid_clock.cc:648] HybridClock initialized: now 1781920406757718 us; error 127 us; skew 500 ppm
I20260620 01:53:26.758400 2180 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:26.761385 2180 webserver.cc:533] Webserver started at http://127.20.1.129:35993/ using document root <none> and password file <none>
I20260620 01:53:26.762387 2180 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:26.762537 2180 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:26.770561 2180 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 01:53:26.775604 2202 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:26.778050 2180 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:53:26.778546 2180 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "923994cacde94cb1ba5837858b51375a"
format_stamp: "Formatted at 2026-06-20 01:53:06 on dist-test-slave-wg14"
I20260620 01:53:26.779639 2180 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:26.806950 2180 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:26.808156 2180 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:26.808554 2180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:26.810276 2180 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:26.813524 2180 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:26.813728 2180 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:26.813905 2180 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:26.813969 2180 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:26.860574 2180 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:36755
I20260620 01:53:26.860656 2314 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:36755 every 8 connection(s)
I20260620 01:53:26.863018 2180 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:53:26.871452 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 2180
I20260620 01:53:26.871768 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:53:26.877979 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:53:26.895855 2315 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33299
I20260620 01:53:26.896412 2315 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:26.897835 2315 heartbeater.cc:507] Master 127.20.1.188:33299 requested a full tablet report, sending...
I20260620 01:53:26.898986 2316 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40263
I20260620 01:53:26.899205 2316 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:26.899928 2316 heartbeater.cc:507] Master 127.20.1.190:40263 requested a full tablet report, sending...
I20260620 01:53:26.901342 2317 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36607
I20260620 01:53:26.901543 2317 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:26.902081 1946 ts_manager.cc:194] Registered new tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:36755)
I20260620 01:53:26.902091 2116 ts_manager.cc:194] Registered new tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:36755)
I20260620 01:53:26.902141 2317 heartbeater.cc:507] Master 127.20.1.189:36607 requested a full tablet report, sending...
I20260620 01:53:26.904803 1946 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:42579
I20260620 01:53:26.905206 2030 ts_manager.cc:194] Registered new tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:36755)
I20260620 01:53:27.172212 2157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "025e1c47b0614610ad9da9addf02bdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:27.173242 2157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: This master's current role is: LEADER
W20260620 01:53:27.335196 2326 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:27.335599 2326 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:27.335741 2326 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:27.346922 2326 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:27.347388 2326 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:53:27.361421 2326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:27.363503 2326 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:27.366034 2326 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:27.385286 2337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:27.385277 2335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:27.385551 2334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:27.385843 2326 server_base.cc:1061] running on GCE node
I20260620 01:53:27.386889 2326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:27.388500 2326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:27.392115 2326 hybrid_clock.cc:648] HybridClock initialized: now 1781920407391930 us; error 51 us; skew 500 ppm
I20260620 01:53:27.392860 2326 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:27.396337 2326 webserver.cc:533] Webserver started at http://127.20.1.130:39777/ using document root <none> and password file <none>
I20260620 01:53:27.397397 2326 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:27.397589 2326 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:27.405368 2326 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 01:53:27.409899 2343 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:27.412272 2326 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:53:27.412524 2326 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "f7e99ead91bc4b29a7a4675f29b8f13c"
format_stamp: "Formatted at 2026-06-20 01:53:06 on dist-test-slave-wg14"
I20260620 01:53:27.413470 2326 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:27.442302 2326 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:27.443495 2326 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:27.443889 2326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:27.445433 2326 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:27.448130 2326 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:27.448272 2326 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:27.448382 2326 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:27.448431 2326 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:27.493746 2326 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:39085
I20260620 01:53:27.493805 2455 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:39085 every 8 connection(s)
I20260620 01:53:27.496095 2326 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:53:27.505492 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 2326
I20260620 01:53:27.505764 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
I20260620 01:53:27.513599 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:53:27.524026 2457 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40263
I20260620 01:53:27.524560 2457 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:27.526086 2457 heartbeater.cc:507] Master 127.20.1.190:40263 requested a full tablet report, sending...
I20260620 01:53:27.528616 1946 ts_manager.cc:194] Registered new tserver with Master: f7e99ead91bc4b29a7a4675f29b8f13c (127.20.1.130:39085)
I20260620 01:53:27.528823 2456 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33299
I20260620 01:53:27.528997 2456 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:27.529467 2456 heartbeater.cc:507] Master 127.20.1.188:33299 requested a full tablet report, sending...
I20260620 01:53:27.529624 1946 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.130:51515
I20260620 01:53:27.530795 2116 ts_manager.cc:194] Registered new tserver with Master: f7e99ead91bc4b29a7a4675f29b8f13c (127.20.1.130:39085)
I20260620 01:53:27.538873 2458 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36607
I20260620 01:53:27.539129 2458 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:27.539672 2458 heartbeater.cc:507] Master 127.20.1.189:36607 requested a full tablet report, sending...
I20260620 01:53:27.541785 2030 ts_manager.cc:194] Registered new tserver with Master: f7e99ead91bc4b29a7a4675f29b8f13c (127.20.1.130:39085)
I20260620 01:53:27.908753 2316 heartbeater.cc:499] Master 127.20.1.190:40263 was elected leader, sending a full tablet report...
W20260620 01:53:27.969627 2466 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:27.970069 2466 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:27.970242 2466 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:27.981536 2466 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:27.981732 2466 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:53:27.993912 2466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:27.996660 2466 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:27.999419 2466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:28.016422 2474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:28.015542 2472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:28.016155 2471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:28.018070 2466 server_base.cc:1061] running on GCE node
I20260620 01:53:28.019712 2466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:28.022599 2466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:28.026252 2466 hybrid_clock.cc:648] HybridClock initialized: now 1781920408026016 us; error 132 us; skew 500 ppm
I20260620 01:53:28.026805 2466 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:28.031030 2466 webserver.cc:533] Webserver started at http://127.20.1.131:44217/ using document root <none> and password file <none>
I20260620 01:53:28.032200 2466 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:28.032403 2466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:28.041229 2466 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.003s
I20260620 01:53:28.046271 2480 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:28.048786 2466 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:53:28.049082 2466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "e654c72cf9a54ddb802846ce8cd48074"
format_stamp: "Formatted at 2026-06-20 01:53:07 on dist-test-slave-wg14"
I20260620 01:53:28.050504 2466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:28.091882 2466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:28.093000 2466 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:28.093335 2466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:28.094686 2466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:28.097199 2466 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:28.097321 2466 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:28.097450 2466 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:28.097522 2466 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:28.148741 2466 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:45353
I20260620 01:53:28.148810 2594 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:45353 every 8 connection(s)
I20260620 01:53:28.150630 2466 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:53:28.158231 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 2466
I20260620 01:53:28.158512 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
I20260620 01:53:28.196012 2595 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33299
I20260620 01:53:28.196188 2596 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40263
I20260620 01:53:28.196532 2596 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:28.198447 2596 heartbeater.cc:507] Master 127.20.1.190:40263 requested a full tablet report, sending...
I20260620 01:53:28.196014 2597 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36607
I20260620 01:53:28.199721 2597 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:28.200485 2597 heartbeater.cc:507] Master 127.20.1.189:36607 requested a full tablet report, sending...
I20260620 01:53:28.196532 2595 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:28.203346 2595 heartbeater.cc:507] Master 127.20.1.188:33299 requested a full tablet report, sending...
I20260620 01:53:28.203533 2030 ts_manager.cc:194] Registered new tserver with Master: e654c72cf9a54ddb802846ce8cd48074 (127.20.1.131:45353)
I20260620 01:53:28.204982 1945 ts_manager.cc:194] Registered new tserver with Master: e654c72cf9a54ddb802846ce8cd48074 (127.20.1.131:45353)
I20260620 01:53:28.206081 2116 ts_manager.cc:194] Registered new tserver with Master: e654c72cf9a54ddb802846ce8cd48074 (127.20.1.131:45353)
I20260620 01:53:28.207362 1945 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.131:52429
I20260620 01:53:28.223218 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:53:28.239972 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:53:28.240271 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 1915
I20260620 01:53:28.242102 1976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:28.369345 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 1915
I20260620 01:53:28.390403 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40263
--webserver_interface=127.20.1.190
--webserver_port=39429
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299 with env {}
I20260620 01:53:28.533483 2457 heartbeater.cc:499] Master 127.20.1.190:40263 was elected leader, sending a full tablet report...
W20260620 01:53:28.535738 2457 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:40263 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:40263: connect: Connection refused (error 111)
W20260620 01:53:28.834686 2606 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:28.835202 2606 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:28.835381 2606 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:28.845875 2606 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:28.846014 2606 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:28.846079 2606 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:28.846150 2606 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:28.857888 2606 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=39429
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:28.860028 2606 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:28.862571 2606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:28.873798 2612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:28.874863 2613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:28.875538 2615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:28.876403 2606 server_base.cc:1061] running on GCE node
I20260620 01:53:28.877954 2606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:28.880398 2606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:28.883948 2606 hybrid_clock.cc:648] HybridClock initialized: now 1781920408883720 us; error 115 us; skew 500 ppm
I20260620 01:53:28.884532 2606 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:28.887701 2606 webserver.cc:533] Webserver started at http://127.20.1.190:39429/ using document root <none> and password file <none>
I20260620 01:53:28.888819 2606 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:28.888978 2606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:28.896631 2606 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:53:28.901885 2621 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:28.904098 2606 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:53:28.904346 2606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "025e1c47b0614610ad9da9addf02bdaf"
format_stamp: "Formatted at 2026-06-20 01:53:03 on dist-test-slave-wg14"
I20260620 01:53:28.905262 2606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260620 01:53:28.914427 2316 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:40263 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:40263: connect: Connection refused (error 111)
I20260620 01:53:28.927904 2606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:28.929054 2606 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:28.929476 2606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:28.956965 2606 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:40263
I20260620 01:53:28.957089 2674 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:40263 every 8 connection(s)
I20260620 01:53:28.959679 2606 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:53:28.969031 2675 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:53:28.969519 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 2606
I20260620 01:53:28.969866 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 2180
I20260620 01:53:28.974298 2675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: Bootstrap starting.
I20260620 01:53:28.985227 2310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:29.001899 2675 log.cc:826] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:29.019376 2675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:53:29.020253 2675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: Bootstrap complete.
I20260620 01:53:29.031009 2675 raft_consensus.cc:359] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:29.032196 2675 raft_consensus.cc:740] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025e1c47b0614610ad9da9addf02bdaf, State: Initialized, Role: FOLLOWER
I20260620 01:53:29.033461 2675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:29.034996 2681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:29.036375 2681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:29.037676 2675 sys_catalog.cc:565] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:53:29.051930 2692 catalog_manager.cc:1294] Loaded cluster ID: 537bf1ccc4844dc3a6fa159d662de766
I20260620 01:53:29.052187 2692 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: loading cluster ID for follower catalog manager: success
I20260620 01:53:29.057535 2692 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: acquiring CA information for follower catalog manager: success
I20260620 01:53:29.062866 2692 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:53:29.140888 2606 master_runner.cc:264] No leader master found from master 025e1c47b0614610ad9da9addf02bdaf
I20260620 01:53:29.141119 2606 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:53:29.141899 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 2180
I20260620 01:53:29.159986 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:36755
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--webserver_port=35993
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:53:29.211385 2596 heartbeater.cc:499] Master 127.20.1.190:40263 was elected leader, sending a full tablet report...
I20260620 01:53:29.247500 2639 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e654c72cf9a54ddb802846ce8cd48074" instance_seqno: 1781920408137745) as {username='slave'} at 127.20.1.131:47485; Asking this server to re-register.
I20260620 01:53:29.249181 2596 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:29.252101 2639 ts_manager.cc:194] Registered new tserver with Master: e654c72cf9a54ddb802846ce8cd48074 (127.20.1.131:45353)
I20260620 01:53:29.508901 2700 raft_consensus.cc:493] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 025e1c47b0614610ad9da9addf02bdaf)
I20260620 01:53:29.509577 2700 raft_consensus.cc:515] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:29.522281 2700 leader_election.cc:290] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 025e1c47b0614610ad9da9addf02bdaf (127.20.1.190:40263), 8781c9b655eb4aeca5d56a909dc1211e (127.20.1.188:33299)
I20260620 01:53:29.525770 2650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b171c7ed0a74499abcaa971b8f349a97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "025e1c47b0614610ad9da9addf02bdaf" is_pre_election: true
I20260620 01:53:29.526746 2650 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b171c7ed0a74499abcaa971b8f349a97 in term 2.
I20260620 01:53:29.528131 2015 leader_election.cc:304] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025e1c47b0614610ad9da9addf02bdaf, b171c7ed0a74499abcaa971b8f349a97; no voters:
I20260620 01:53:29.529014 2700 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 01:53:29.529276 2700 raft_consensus.cc:493] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 025e1c47b0614610ad9da9addf02bdaf)
I20260620 01:53:29.529608 2700 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:53:29.533541 2700 raft_consensus.cc:515] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:29.548741 2125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b171c7ed0a74499abcaa971b8f349a97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8781c9b655eb4aeca5d56a909dc1211e" is_pre_election: true
I20260620 01:53:29.549094 2650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b171c7ed0a74499abcaa971b8f349a97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "025e1c47b0614610ad9da9addf02bdaf"
I20260620 01:53:29.549346 2650 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:53:29.550570 2700 leader_election.cc:290] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [CANDIDATE]: Term 3 election: Requested vote from peers 025e1c47b0614610ad9da9addf02bdaf (127.20.1.190:40263), 8781c9b655eb4aeca5d56a909dc1211e (127.20.1.188:33299)
I20260620 01:53:29.552095 2125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b171c7ed0a74499abcaa971b8f349a97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8781c9b655eb4aeca5d56a909dc1211e"
I20260620 01:53:29.560971 2650 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b171c7ed0a74499abcaa971b8f349a97 in term 3.
I20260620 01:53:29.562094 2015 leader_election.cc:304] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 025e1c47b0614610ad9da9addf02bdaf, b171c7ed0a74499abcaa971b8f349a97; no voters: 8781c9b655eb4aeca5d56a909dc1211e
I20260620 01:53:29.562942 2700 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 01:53:29.565021 2700 raft_consensus.cc:697] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [term 3 LEADER]: Becoming Leader. State: Replica: b171c7ed0a74499abcaa971b8f349a97, State: Running, Role: LEADER
I20260620 01:53:29.566071 2700 consensus_queue.cc:237] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:29.568853 2457 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40263
I20260620 01:53:29.570117 2457 heartbeater.cc:499] Master 127.20.1.190:40263 was elected leader, sending a full tablet report...
I20260620 01:53:29.572489 2639 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f7e99ead91bc4b29a7a4675f29b8f13c" instance_seqno: 1781920407482324) as {username='slave'} at 127.20.1.130:38145; Asking this server to re-register.
I20260620 01:53:29.573829 2457 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:29.575815 2639 ts_manager.cc:194] Registered new tserver with Master: f7e99ead91bc4b29a7a4675f29b8f13c (127.20.1.130:39085)
I20260620 01:53:29.579473 2704 sys_catalog.cc:455] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b171c7ed0a74499abcaa971b8f349a97. Latest consensus state: current_term: 3 leader_uuid: "b171c7ed0a74499abcaa971b8f349a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:29.580008 2704 sys_catalog.cc:458] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:29.581113 2706 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:53:29.584462 2706 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:53:29.586395 2706 catalog_manager.cc:1294] Loaded cluster ID: 537bf1ccc4844dc3a6fa159d662de766
I20260620 01:53:29.586709 2706 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:53:29.589233 2706 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:53:29.590775 2706 catalog_manager.cc:6080] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97: Loaded TSK: 0
I20260620 01:53:29.592455 2706 catalog_manager.cc:1549] Initializing in-progress tserver states...
W20260620 01:53:29.671921 2698 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:29.672331 2698 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:29.672443 2698 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:29.682786 2698 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:29.683012 2698 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:53:29.696521 2698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:36755
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=35993
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:29.698310 2698 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:29.701467 2698 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:29.716718 2711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:29.716899 2714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:29.716825 2712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:29.718192 2698 server_base.cc:1061] running on GCE node
I20260620 01:53:29.719025 2698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:29.719893 2716 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 025e1c47b0614610ad9da9addf02bdaf)
I20260620 01:53:29.720153 2716 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } }
I20260620 01:53:29.720975 2698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:29.723383 2716 leader_election.cc:290] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 025e1c47b0614610ad9da9addf02bdaf (127.20.1.190:40263), b171c7ed0a74499abcaa971b8f349a97 (127.20.1.189:36607)
I20260620 01:53:29.724215 2041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8781c9b655eb4aeca5d56a909dc1211e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b171c7ed0a74499abcaa971b8f349a97" is_pre_election: true
I20260620 01:53:29.724538 2698 hybrid_clock.cc:648] HybridClock initialized: now 1781920409724266 us; error 172 us; skew 500 ppm
I20260620 01:53:29.725081 2698 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:29.730831 2698 webserver.cc:533] Webserver started at http://127.20.1.129:35993/ using document root <none> and password file <none>
I20260620 01:53:29.733511 2698 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:29.733834 2698 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:29.735280 2650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8781c9b655eb4aeca5d56a909dc1211e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "025e1c47b0614610ad9da9addf02bdaf" is_pre_election: true
I20260620 01:53:29.736019 2650 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8781c9b655eb4aeca5d56a909dc1211e in current term 3: Already voted for candidate b171c7ed0a74499abcaa971b8f349a97 in this term.
I20260620 01:53:29.737324 2101 leader_election.cc:304] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8781c9b655eb4aeca5d56a909dc1211e; no voters: 025e1c47b0614610ad9da9addf02bdaf, b171c7ed0a74499abcaa971b8f349a97
I20260620 01:53:29.737969 2716 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:53:29.741144 2716 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 01:53:29.743562 2698 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.003s sys 0.004s
I20260620 01:53:29.748762 2722 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:29.751533 2698 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260620 01:53:29.752455 2698 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "923994cacde94cb1ba5837858b51375a"
format_stamp: "Formatted at 2026-06-20 01:53:06 on dist-test-slave-wg14"
I20260620 01:53:29.753657 2698 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:29.777137 2698 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:29.778247 2698 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:29.778586 2698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:29.779852 2698 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:29.782238 2698 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:29.782366 2698 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:29.782490 2698 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:29.782527 2698 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:29.826649 2698 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:36755
I20260620 01:53:29.827067 2837 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:36755 every 8 connection(s)
I20260620 01:53:29.828680 2698 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:53:29.837352 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 2698
I20260620 01:53:29.880671 2840 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36607
I20260620 01:53:29.881264 2840 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:29.882831 2840 heartbeater.cc:507] Master 127.20.1.189:36607 requested a full tablet report, sending...
I20260620 01:53:29.885411 2839 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40263
I20260620 01:53:29.885639 2839 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:29.886045 2838 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33299
I20260620 01:53:29.886236 2838 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:29.886332 2029 ts_manager.cc:194] Re-registered known tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:36755)
I20260620 01:53:29.886430 2839 heartbeater.cc:507] Master 127.20.1.190:40263 requested a full tablet report, sending...
I20260620 01:53:29.886909 2838 heartbeater.cc:507] Master 127.20.1.188:33299 requested a full tablet report, sending...
I20260620 01:53:29.888638 2639 ts_manager.cc:194] Registered new tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:36755)
I20260620 01:53:29.889667 2029 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:35123
I20260620 01:53:29.890288 2116 ts_manager.cc:194] Re-registered known tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:36755)
I20260620 01:53:29.999558 2125 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [term 3 FOLLOWER]: Refusing update from remote peer b171c7ed0a74499abcaa971b8f349a97: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:53:30.001011 2704 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:53:30.016546 2716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b171c7ed0a74499abcaa971b8f349a97. Latest consensus state: current_term: 3 leader_uuid: "b171c7ed0a74499abcaa971b8f349a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:30.016938 2716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:30.043239 2700 sys_catalog.cc:455] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b171c7ed0a74499abcaa971b8f349a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:30.043632 2700 sys_catalog.cc:458] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:30.045639 2650 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [term 3 FOLLOWER]: Refusing update from remote peer b171c7ed0a74499abcaa971b8f349a97: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:53:30.046201 2716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b171c7ed0a74499abcaa971b8f349a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:30.047166 2716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8781c9b655eb4aeca5d56a909dc1211e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:30.047479 2700 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:53:30.052392 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 2698
I20260620 01:53:30.076990 2852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b171c7ed0a74499abcaa971b8f349a97. Latest consensus state: current_term: 3 leader_uuid: "b171c7ed0a74499abcaa971b8f349a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:30.077427 2852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:30.077790 2857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b171c7ed0a74499abcaa971b8f349a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:30.078101 2857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025e1c47b0614610ad9da9addf02bdaf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:30.083649 2832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:30.192968 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 2698
I20260620 01:53:30.217613 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:36755
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--webserver_port=35993
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
I20260620 01:53:30.483345 2851 sys_catalog.cc:455] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b171c7ed0a74499abcaa971b8f349a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e1c47b0614610ad9da9addf02bdaf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40263 } } peers { permanent_uuid: "b171c7ed0a74499abcaa971b8f349a97" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36607 } } peers { permanent_uuid: "8781c9b655eb4aeca5d56a909dc1211e" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33299 } } }
I20260620 01:53:30.483702 2851 sys_catalog.cc:458] T 00000000000000000000000000000000 P b171c7ed0a74499abcaa971b8f349a97 [sys.catalog]: This master's current role is: LEADER
W20260620 01:53:30.665388 2859 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:30.665864 2859 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:30.665990 2859 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:53:30.666054 2859 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:30.677925 2859 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:30.678262 2859 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:53:30.692014 2859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:36755
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=35993
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:30.693943 2859 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:30.696117 2859 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:30.708665 2865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:30.709493 2864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:30.709636 2867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:30.710081 2859 server_base.cc:1061] running on GCE node
I20260620 01:53:30.710878 2859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:30.712499 2859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:30.716053 2859 hybrid_clock.cc:648] HybridClock initialized: now 1781920410715785 us; error 130 us; skew 500 ppm
I20260620 01:53:30.716605 2859 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:30.720858 2859 webserver.cc:533] Webserver started at http://127.20.1.129:35993/ using document root <none> and password file <none>
I20260620 01:53:30.722029 2859 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:30.722208 2859 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:30.731776 2859 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 01:53:30.736618 2873 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:30.738957 2859 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:53:30.739275 2859 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "923994cacde94cb1ba5837858b51375a"
format_stamp: "Formatted at 2026-06-20 01:53:06 on dist-test-slave-wg14"
I20260620 01:53:30.740453 2859 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:30.756253 2859 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:30.757335 2859 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:30.757644 2859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:30.758872 2859 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:30.761132 2859 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:30.761271 2859 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:30.761499 2859 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:30.761670 2859 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:53:30.762229 2859 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:53:30.762338 2859 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:53:30.762421 2859 fault_injection.cc:59] injecting fault for kudu (pid 2859): FLAGS_fault_before_start (process will exit)
I20260620 01:53:30.779448 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:36755
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--webserver_port=35993
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:53:31.217641 2597 heartbeater.cc:499] Master 127.20.1.189:36607 was elected leader, sending a full tablet report...
W20260620 01:53:31.293035 2881 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:31.293432 2881 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:31.293529 2881 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:31.304929 2881 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:31.305241 2881 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:53:31.317952 2881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41301,127.20.1.149:42539,127.20.1.150:36487,127.20.1.151:45281,127.20.1.152:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:36755
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=35993
--tserver_master_addrs=127.20.1.190:40263,127.20.1.189:36607,127.20.1.188:33299
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:31.320070 2881 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:31.322952 2881 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:31.337230 2887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:31.337697 2888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:31.337895 2890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:31.339157 2881 server_base.cc:1061] running on GCE node
I20260620 01:53:31.340284 2881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:31.342437 2881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:31.346143 2881 hybrid_clock.cc:648] HybridClock initialized: now 1781920411345935 us; error 110 us; skew 500 ppm
I20260620 01:53:31.346771 2881 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:53:31.350920 2881 webserver.cc:533] Webserver started at http://127.20.1.129:35993/ using document root <none> and password file <none>
I20260620 01:53:31.352108 2881 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:31.352625 2881 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:31.360911 2881 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:53:31.365556 2896 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:31.367830 2881 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260620 01:53:31.368090 2881 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "923994cacde94cb1ba5837858b51375a"
format_stamp: "Formatted at 2026-06-20 01:53:06 on dist-test-slave-wg14"
I20260620 01:53:31.370103 2881 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:31.392760 2881 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:31.393901 2881 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:31.394284 2881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:31.395638 2881 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:31.398298 2881 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:31.398427 2881 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:31.398794 2881 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:31.398936 2881 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:31.445142 2881 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:36755
I20260620 01:53:31.445384 3010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:36755 every 8 connection(s)
I20260620 01:53:31.447016 2881 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:53:31.453001 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 2881
I20260620 01:53:31.453356 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 2881
I20260620 01:53:31.489954 3011 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33299
I20260620 01:53:31.490538 3011 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:31.491184 3012 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40263
I20260620 01:53:31.491395 3012 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:31.492362 3012 heartbeater.cc:507] Master 127.20.1.190:40263 requested a full tablet report, sending...
I20260620 01:53:31.492648 3011 heartbeater.cc:507] Master 127.20.1.188:33299 requested a full tablet report, sending...
I20260620 01:53:31.495257 2639 ts_manager.cc:194] Re-registered known tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:36755)
I20260620 01:53:31.495838 3006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:31.498522 2116 ts_manager.cc:194] Re-registered known tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:36755)
I20260620 01:53:31.555315 2458 heartbeater.cc:499] Master 127.20.1.189:36607 was elected leader, sending a full tablet report...
I20260620 01:53:31.604058 3013 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36607
I20260620 01:53:31.604533 3013 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:31.605472 3013 heartbeater.cc:507] Master 127.20.1.189:36607 requested a full tablet report, sending...
I20260620 01:53:31.606391 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 2881
I20260620 01:53:31.607005 2029 ts_manager.cc:194] Re-registered known tserver with Master: 923994cacde94cb1ba5837858b51375a (127.20.1.129:36755)
I20260620 01:53:31.610117 2029 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:36255
W20260620 01:53:31.628144 2014 connection.cc:573] server connection from 127.20.1.129:36255 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:53:31.629065 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 2326
I20260620 01:53:31.643743 2451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:31.753057 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 2326
I20260620 01:53:31.769109 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 2466
I20260620 01:53:31.780457 2590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:31.882036 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 2466
I20260620 01:53:31.900434 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 2606
I20260620 01:53:31.909723 2670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:32.038897 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 2606
I20260620 01:53:32.061949 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 1983
I20260620 01:53:32.063856 2061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:32.191218 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 1983
W20260620 01:53:32.213372 2100 connection.cc:573] server connection from 127.0.0.1:60110 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:53:32.213974 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 2068
I20260620 01:53:32.215721 2146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:32.338886 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 2068
Shutting down hive metastore.
2026-06-20T01:53:32Z chronyd exiting
2026-06-20T01:53:32Z chronyd exiting
2026-06-20T01:53:32Z chronyd exiting
2026-06-20T01:53:32Z chronyd exiting
2026-06-20T01:53:32Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/5 (50766 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/6
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 01:53:32 dist-test-slave-wg14 krb5kdc[3038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 01:53:32 dist-test-slave-wg14 krb5kdc[3038](info): set up 2 sockets
Jun 20 01:53:32 dist-test-slave-wg14 krb5kdc[3038](info): commencing operation
krb5kdc: starting...
W20260620 01:53:34.939630 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.111s user 0.008s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 01:53:35 dist-test-slave-wg14 krb5kdc[3038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:35.232676 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:38865
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:53:35.699532 3052 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:35.699965 3052 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:35.700124 3052 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:35.713655 3052 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:35.713824 3052 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:35.713907 3052 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:35.713972 3052 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:35.730607 3052 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:38865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:35.732756 3052 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:35.734516 3052 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:35.746429 3058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:35.747761 3057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:35.748101 3060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:35.748265 3052 server_base.cc:1061] running on GCE node
I20260620 01:53:35.749388 3052 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:35.749639 3052 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:35.749749 3052 hybrid_clock.cc:648] HybridClock initialized: now 1781920415749719 us; error 0 us; skew 500 ppm
I20260620 01:53:35.750204 3052 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:35 dist-test-slave-wg14 krb5kdc[3038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920415, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:35.754973 3052 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:53:35.757748 3052 webserver.cc:533] Webserver started at http://127.20.1.190:36535/ using document root <none> and password file <none>
I20260620 01:53:35.758671 3052 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:35.758805 3052 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:35.759280 3052 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:53:35.763206 3052 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "a74f7ecbb99a46c697110c241b68fe79"
format_stamp: "Formatted at 2026-06-20 01:53:35 on dist-test-slave-wg14"
I20260620 01:53:35.764801 3052 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "a74f7ecbb99a46c697110c241b68fe79"
format_stamp: "Formatted at 2026-06-20 01:53:35 on dist-test-slave-wg14"
I20260620 01:53:35.774042 3052 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20260620 01:53:35.779574 3066 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:35.781816 3052 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260620 01:53:35.782079 3052 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "a74f7ecbb99a46c697110c241b68fe79"
format_stamp: "Formatted at 2026-06-20 01:53:35 on dist-test-slave-wg14"
I20260620 01:53:35.782385 3052 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:35.808706 3052 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:35.815834 3052 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:35.816203 3052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:35.849586 3052 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:38865
I20260620 01:53:35.849655 3117 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:38865 every 8 connection(s)
I20260620 01:53:35.852016 3052 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:53:35.855773 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 3052
I20260620 01:53:35.855995 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:53:35.858937 3118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:53:35.871603 3118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.189@KRBTEST.COM" created.
Jun 20 01:53:35 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920415, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:53:35.911242 3118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:53:35.913635 3070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:41021: connect: Connection refused (error 111)
W20260620 01:53:35.916093 3118 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:41021: Network error: Client connection negotiation failed: client connection to 127.20.1.189:41021: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.189@KRBTEST.COM" created.
I20260620 01:53:35.965780 3118 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } attempt: 1
W20260620 01:53:35.969408 3118 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:41021: Network error: Client connection negotiation failed: client connection to 127.20.1.189:41021: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:35.980857 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:41021
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:53:36.037971 3118 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } attempt: 2
W20260620 01:53:36.041337 3118 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:41021: Network error: Client connection negotiation failed: client connection to 127.20.1.189:41021: connect: Connection refused (error 111)
I20260620 01:53:36.132903 3118 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } attempt: 3
W20260620 01:53:36.136149 3118 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:41021: Network error: Client connection negotiation failed: client connection to 127.20.1.189:41021: connect: Connection refused (error 111)
I20260620 01:53:36.279510 3118 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } attempt: 4
W20260620 01:53:36.283363 3118 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:41021: Network error: Client connection negotiation failed: client connection to 127.20.1.189:41021: connect: Connection refused (error 111)
W20260620 01:53:36.471529 3127 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:36.471940 3127 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:36.472090 3127 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:36.483100 3127 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:36.483426 3127 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:36.483515 3127 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:36.483570 3127 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:36.498346 3127 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:41021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:36.500846 3127 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:36.504024 3127 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:36.517680 3132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:36.517850 3134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:36.519002 3136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:36.519742 3127 server_base.cc:1061] running on GCE node
I20260620 01:53:36.521311 3127 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:36.521535 3127 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:36.521631 3127 hybrid_clock.cc:648] HybridClock initialized: now 1781920416521606 us; error 0 us; skew 500 ppm
I20260620 01:53:36.522379 3127 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:36 dist-test-slave-wg14 krb5kdc[3038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920416, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:36.528434 3127 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 01:53:36.531689 3127 webserver.cc:533] Webserver started at http://127.20.1.189:40045/ using document root <none> and password file <none>
I20260620 01:53:36.532699 3127 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:36.532832 3127 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:36.533237 3127 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:53:36.536330 3127 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "0d2963ab3b994f0785ed4a2e74c8d616"
format_stamp: "Formatted at 2026-06-20 01:53:36 on dist-test-slave-wg14"
I20260620 01:53:36.537832 3127 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "0d2963ab3b994f0785ed4a2e74c8d616"
format_stamp: "Formatted at 2026-06-20 01:53:36 on dist-test-slave-wg14"
I20260620 01:53:36.548161 3127 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s
I20260620 01:53:36.554741 3142 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:36.557700 3127 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:53:36.558000 3127 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "0d2963ab3b994f0785ed4a2e74c8d616"
format_stamp: "Formatted at 2026-06-20 01:53:36 on dist-test-slave-wg14"
I20260620 01:53:36.558280 3127 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:36.583030 3118 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } attempt: 5
W20260620 01:53:36.585764 3118 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:41021: Network error: Client connection negotiation failed: client connection to 127.20.1.189:41021: connect: Connection refused (error 111)
I20260620 01:53:36.616946 3127 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:36.629845 3127 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:36.630224 3127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:36.670043 3127 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:41021
I20260620 01:53:36.670066 3193 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:41021 every 8 connection(s)
I20260620 01:53:36.672133 3127 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:53:36.678408 3194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:53:36.681799 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 3127
I20260620 01:53:36.682104 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:53:36.685672 3194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
Jun 20 01:53:36 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920416, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.188@KRBTEST.COM" created.
I20260620 01:53:36.721890 3194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:53:36 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920416, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:53:36.749277 3194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:53:36.751648 3144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33775: connect: Connection refused (error 111)
W20260620 01:53:36.754863 3194 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33775: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33775: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.188@KRBTEST.COM" created.
I20260620 01:53:36.804471 3194 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } attempt: 1
W20260620 01:53:36.808192 3194 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33775: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33775: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:36.816371 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:33775
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:53:36.876634 3194 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } attempt: 2
W20260620 01:53:36.879710 3194 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33775: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33775: connect: Connection refused (error 111)
I20260620 01:53:36.971066 3194 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } attempt: 3
W20260620 01:53:36.973730 3194 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33775: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33775: connect: Connection refused (error 111)
I20260620 01:53:37.117092 3194 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } attempt: 4
W20260620 01:53:37.120123 3194 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33775: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33775: connect: Connection refused (error 111)
I20260620 01:53:37.133209 3118 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } attempt: 6
Jun 20 01:53:37 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920415, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:53:37.154490 3118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:53:37.157917 3118 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33775: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33775: connect: Connection refused (error 111)
I20260620 01:53:37.210235 3118 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } attempt: 1
W20260620 01:53:37.212874 3118 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33775: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33775: connect: Connection refused (error 111)
W20260620 01:53:37.283317 3204 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:37.283751 3204 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:37.283927 3204 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260620 01:53:37.287241 3118 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } attempt: 2
W20260620 01:53:37.290527 3118 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33775: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33775: connect: Connection refused (error 111)
W20260620 01:53:37.296476 3204 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:37.296687 3204 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:37.296819 3204 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:37.296896 3204 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:37.310559 3204 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:33775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:37.312775 3204 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:37.315531 3204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:37.330392 3211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:37.330407 3212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:37.330801 3214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:37.331722 3204 server_base.cc:1061] running on GCE node
I20260620 01:53:37.332656 3204 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:37.332890 3204 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:37.332984 3204 hybrid_clock.cc:648] HybridClock initialized: now 1781920417332957 us; error 0 us; skew 500 ppm
I20260620 01:53:37.333352 3204 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:37 dist-test-slave-wg14 krb5kdc[3038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920417, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:37.338008 3204 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 01:53:37.341646 3204 webserver.cc:533] Webserver started at http://127.20.1.188:44731/ using document root <none> and password file <none>
I20260620 01:53:37.342941 3204 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:37.343099 3204 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:37.344007 3204 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:53:37.348315 3204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "f007e88279d847129be498cca804b3af"
format_stamp: "Formatted at 2026-06-20 01:53:37 on dist-test-slave-wg14"
I20260620 01:53:37.349845 3204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "f007e88279d847129be498cca804b3af"
format_stamp: "Formatted at 2026-06-20 01:53:37 on dist-test-slave-wg14"
I20260620 01:53:37.358266 3204 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260620 01:53:37.364776 3220 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:37.366863 3204 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:53:37.367113 3204 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "f007e88279d847129be498cca804b3af"
format_stamp: "Formatted at 2026-06-20 01:53:37 on dist-test-slave-wg14"
I20260620 01:53:37.367393 3204 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:37.403880 3118 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } attempt: 3
W20260620 01:53:37.407353 3118 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33775: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33775: connect: Connection refused (error 111)
I20260620 01:53:37.410985 3204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:37.416174 3204 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:37.416512 3204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:37.419494 3194 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } attempt: 5
W20260620 01:53:37.422312 3194 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:33775: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33775: connect: Connection refused (error 111)
I20260620 01:53:37.447650 3204 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:33775
I20260620 01:53:37.447705 3271 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:33775 every 8 connection(s)
I20260620 01:53:37.450341 3204 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:53:37.457409 3272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:53:37.460824 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 3204
I20260620 01:53:37.461124 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:53:37.465338 3272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:53:37 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:53:37 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920417, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:53:37 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:53:37.511304 3272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:53:37 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920417, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:53:37 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:53:37.534236 3272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:53:37 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920417, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:53:37.556887 3118 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } attempt: 4
I20260620 01:53:37.564746 3272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af: Bootstrap starting.
Jun 20 01:53:37 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920415, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:53:37.571334 3272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af: Neither blocks nor log segments found. Creating new log.
I20260620 01:53:37.573858 3272 log.cc:826] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:37.578619 3272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af: No bootstrap required, opened a new log
I20260620 01:53:37.586882 3272 raft_consensus.cc:359] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:37.587404 3272 raft_consensus.cc:385] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:53:37.587580 3272 raft_consensus.cc:740] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f007e88279d847129be498cca804b3af, State: Initialized, Role: FOLLOWER
I20260620 01:53:37.587677 3118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: Bootstrap starting.
I20260620 01:53:37.588595 3272 consensus_queue.cc:260] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:37.590057 3282 sys_catalog.cc:455] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:37.590497 3282 sys_catalog.cc:458] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:37.591953 3272 sys_catalog.cc:565] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:53:37.592377 3118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: Neither blocks nor log segments found. Creating new log.
I20260620 01:53:37.593830 3118 log.cc:826] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:37.601619 3118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: No bootstrap required, opened a new log
I20260620 01:53:37.611791 3118 raft_consensus.cc:359] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:37.612308 3118 raft_consensus.cc:385] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:53:37.612465 3118 raft_consensus.cc:740] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74f7ecbb99a46c697110c241b68fe79, State: Initialized, Role: FOLLOWER
W20260620 01:53:37.612425 3294 catalog_manager.cc:1593] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:53:37.612757 3294 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:53:37.613739 3118 consensus_queue.cc:260] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:37.620115 3296 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:37.620697 3296 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:37.609911 3204 master_runner.cc:187] Error getting master registration for 127.20.1.190:38865: OK, instance_id { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" instance_seqno: 1781920415841388 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:53:37.621716 3204 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:53:37.621963 3118 sys_catalog.cc:565] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:53:37.639422 3307 catalog_manager.cc:1593] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:53:37.639824 3307 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:53:37.637655 3052 master_runner.cc:187] Error getting master registration for 127.20.1.189:41021: OK, instance_id { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" instance_seqno: 1781920416656137 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:53:37.645579 3052 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:53:37.669325 3226 tablet.cc:2406] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:53:37.701789 3282 raft_consensus.cc:493] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:53:37.702260 3282 raft_consensus.cc:515] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:37.706889 3093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f007e88279d847129be498cca804b3af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74f7ecbb99a46c697110c241b68fe79" is_pre_election: true
I20260620 01:53:37.707736 3093 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f007e88279d847129be498cca804b3af in term 0.
I20260620 01:53:37.708382 3282 leader_election.cc:290] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a74f7ecbb99a46c697110c241b68fe79 (127.20.1.190:38865), 0d2963ab3b994f0785ed4a2e74c8d616 (127.20.1.189:41021)
I20260620 01:53:37.709116 3221 leader_election.cc:304] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74f7ecbb99a46c697110c241b68fe79, f007e88279d847129be498cca804b3af; no voters:
I20260620 01:53:37.709784 3282 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:53:37.709965 3282 raft_consensus.cc:493] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:53:37.710114 3282 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:53:37.713846 3282 raft_consensus.cc:515] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:37.715344 3282 leader_election.cc:290] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [CANDIDATE]: Term 1 election: Requested vote from peers a74f7ecbb99a46c697110c241b68fe79 (127.20.1.190:38865), 0d2963ab3b994f0785ed4a2e74c8d616 (127.20.1.189:41021)
I20260620 01:53:37.715984 3093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f007e88279d847129be498cca804b3af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74f7ecbb99a46c697110c241b68fe79"
I20260620 01:53:37.716468 3093 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:53:37.720862 3093 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f007e88279d847129be498cca804b3af in term 1.
I20260620 01:53:37.721951 3221 leader_election.cc:304] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74f7ecbb99a46c697110c241b68fe79, f007e88279d847129be498cca804b3af; no voters:
I20260620 01:53:37.722841 3282 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:53:37.729154 3282 raft_consensus.cc:697] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 1 LEADER]: Becoming Leader. State: Replica: f007e88279d847129be498cca804b3af, State: Running, Role: LEADER
I20260620 01:53:37.731582 3282 consensus_queue.cc:237] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:37.741866 3309 sys_catalog.cc:455] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: SysCatalogTable state changed. Reason: New leader f007e88279d847129be498cca804b3af. Latest consensus state: current_term: 1 leader_uuid: "f007e88279d847129be498cca804b3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:37.742364 3309 sys_catalog.cc:458] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:37.755802 3311 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:53:37.759109 3311 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:53:37.782418 3093 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 1 FOLLOWER]: Refusing update from remote peer f007e88279d847129be498cca804b3af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:53:37.784024 3282 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:53:37.798488 3296 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f007e88279d847129be498cca804b3af. Latest consensus state: current_term: 1 leader_uuid: "f007e88279d847129be498cca804b3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:37.798893 3296 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:37.806710 3312 mvcc.cc:204] Tried to move back new op lower bound from 7298746031221387264 to 7298746031046057984. Current Snapshot: MvccSnapshot[applied={T|T < 7298746031221387264}]
I20260620 01:53:37.809728 3309 sys_catalog.cc:455] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f007e88279d847129be498cca804b3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:37.810355 3309 sys_catalog.cc:458] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:37.814639 3311 catalog_manager.cc:1382] Generated new cluster ID: ec16e38729174834a4f3419529641a3e
I20260620 01:53:37.815111 3311 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:53:37.817380 3296 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f007e88279d847129be498cca804b3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:37.817763 3296 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:37.879601 3311 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:53:37.881990 3311 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:53:37.910548 3311 catalog_manager.cc:6069] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af: Generated new TSK 0
I20260620 01:53:37.913972 3311 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:53:37.969578 3194 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } attempt: 6
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Jun 20 01:53:37 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920416, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:38.032258 3194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616: Bootstrap starting.
I20260620 01:53:38.045979 3194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616: Neither blocks nor log segments found. Creating new log.
I20260620 01:53:38.057192 3194 log.cc:826] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:38.069550 3194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616: No bootstrap required, opened a new log
I20260620 01:53:38.079919 3194 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:38.080536 3194 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:53:38.080879 3194 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2963ab3b994f0785ed4a2e74c8d616, State: Initialized, Role: FOLLOWER
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
I20260620 01:53:38.082319 3194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
I20260620 01:53:38.090245 3322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:38.090675 3322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:38.093997 3194 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:53:38.095412 3167 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:53:38.094192 3168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f007e88279d847129be498cca804b3af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d2963ab3b994f0785ed4a2e74c8d616"
I20260620 01:53:38.094066 3169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f007e88279d847129be498cca804b3af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" is_pre_election: true
I20260620 01:53:38.110782 3167 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 1 FOLLOWER]: Refusing update from remote peer f007e88279d847129be498cca804b3af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:53:38.112290 3282 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 01:53:38.134554 3148 tablet.cc:2406] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:53:38.177649 3331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f007e88279d847129be498cca804b3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:38.178471 3322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f007e88279d847129be498cca804b3af. Latest consensus state: current_term: 1 leader_uuid: "f007e88279d847129be498cca804b3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:38.178793 3322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:38.179046 3331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: This master's current role is: FOLLOWER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:53:38.199947 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.167s user 0.001s sys 0.018s
I20260620 01:53:38.201494 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:53:38.216945 3339 catalog_manager.cc:1294] Loaded cluster ID: ec16e38729174834a4f3419529641a3e
I20260620 01:53:38.217190 3339 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616: loading cluster ID for follower catalog manager: success
I20260620 01:53:38.222628 3339 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616: acquiring CA information for follower catalog manager: success
I20260620 01:53:38.227504 3339 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:53:38.532651 3282 sys_catalog.cc:455] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f007e88279d847129be498cca804b3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:38.533063 3282 sys_catalog.cc:458] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:38.642900 3307 catalog_manager.cc:1294] Loaded cluster ID: ec16e38729174834a4f3419529641a3e
I20260620 01:53:38.643066 3307 catalog_manager.cc:1587] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: loading cluster ID for follower catalog manager: success
I20260620 01:53:38.647764 3307 catalog_manager.cc:1609] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: acquiring CA information for follower catalog manager: success
I20260620 01:53:38.651660 3307 catalog_manager.cc:1637] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 01:53:38.745738 3341 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:38.746143 3341 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:38.746302 3341 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:38.758133 3341 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:38.758420 3341 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:53:38.773914 3341 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:38.776669 3341 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:38.778664 3341 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:38.794209 3349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:38.794327 3352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:38.794260 3350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:38.796056 3341 server_base.cc:1061] running on GCE node
I20260620 01:53:38.797053 3341 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:38.797328 3341 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:38.797449 3341 hybrid_clock.cc:648] HybridClock initialized: now 1781920418797422 us; error 0 us; skew 500 ppm
I20260620 01:53:38.797839 3341 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:38 dist-test-slave-wg14 krb5kdc[3038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920418, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:38.802181 3341 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:53:38.804559 3341 webserver.cc:533] Webserver started at http://127.20.1.129:44371/ using document root <none> and password file <none>
I20260620 01:53:38.805578 3341 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:38.805747 3341 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:38.806150 3341 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:53:38.809137 3341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "024acbbf3d174b4cbe7bdee0c71b0439"
format_stamp: "Formatted at 2026-06-20 01:53:38 on dist-test-slave-wg14"
I20260620 01:53:38.810103 3341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "024acbbf3d174b4cbe7bdee0c71b0439"
format_stamp: "Formatted at 2026-06-20 01:53:38 on dist-test-slave-wg14"
I20260620 01:53:38.818128 3341 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.003s
I20260620 01:53:38.823838 3358 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:38.826208 3341 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260620 01:53:38.826550 3341 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "024acbbf3d174b4cbe7bdee0c71b0439"
format_stamp: "Formatted at 2026-06-20 01:53:38 on dist-test-slave-wg14"
I20260620 01:53:38.826820 3341 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:38.869150 3341 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:38.875816 3341 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:38.876405 3341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:38.877959 3341 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:38.880246 3341 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:38.880364 3341 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:38.880473 3341 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:38.880527 3341 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:38.922370 3341 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:36597
I20260620 01:53:38.922423 3471 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:36597 every 8 connection(s)
I20260620 01:53:38.924203 3341 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:53:38.929587 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 3341
I20260620 01:53:38.929824 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 01:53:38 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920418, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:53:38 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920418, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:53:38 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920418, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.130@KRBTEST.COM; defaulting to no policy
I20260620 01:53:38.978649 3474 heartbeater.cc:344] Connected to a master server at 127.20.1.189:41021
I20260620 01:53:38.979314 3474 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:38.979281 3472 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33775
I20260620 01:53:38.979327 3473 heartbeater.cc:344] Connected to a master server at 127.20.1.190:38865
I20260620 01:53:38.979616 3472 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:38.979667 3473 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:38.980935 3472 heartbeater.cc:507] Master 127.20.1.188:33775 requested a full tablet report, sending...
I20260620 01:53:38.981068 3473 heartbeater.cc:507] Master 127.20.1.190:38865 requested a full tablet report, sending...
I20260620 01:53:38.981112 3474 heartbeater.cc:507] Master 127.20.1.189:41021 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.130@KRBTEST.COM" created.
I20260620 01:53:38.984673 3237 ts_manager.cc:194] Registered new tserver with Master: 024acbbf3d174b4cbe7bdee0c71b0439 (127.20.1.129:36597)
I20260620 01:53:38.985328 3082 ts_manager.cc:194] Registered new tserver with Master: 024acbbf3d174b4cbe7bdee0c71b0439 (127.20.1.129:36597)
I20260620 01:53:38.986894 3237 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:51613
I20260620 01:53:38.989512 3159 ts_manager.cc:194] Registered new tserver with Master: 024acbbf3d174b4cbe7bdee0c71b0439 (127.20.1.129:36597)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:39.076623 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:53:39.516569 3486 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:39.516975 3486 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:39.517279 3486 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:39.527966 3486 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:39.528280 3486 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:53:39.541528 3486 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:39.543916 3486 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:39.545578 3486 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:39.559340 3493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:39.559855 3495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:39.559326 3492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:39.561466 3486 server_base.cc:1061] running on GCE node
I20260620 01:53:39.562387 3486 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:39.562604 3486 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:39.562745 3486 hybrid_clock.cc:648] HybridClock initialized: now 1781920419562713 us; error 0 us; skew 500 ppm
I20260620 01:53:39.563272 3486 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:39 dist-test-slave-wg14 krb5kdc[3038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920419, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:39.570490 3486 init.cc:377] Logged in from keytab as kudu/127.20.1.130@KRBTEST.COM (short username kudu)
I20260620 01:53:39.574043 3486 webserver.cc:533] Webserver started at http://127.20.1.130:41461/ using document root <none> and password file <none>
I20260620 01:53:39.576202 3486 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:39.576498 3486 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:39.577128 3486 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:53:39.581696 3486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/data/instance:
uuid: "5d40809d9a7e4e6c8f043333bbaaab17"
format_stamp: "Formatted at 2026-06-20 01:53:39 on dist-test-slave-wg14"
I20260620 01:53:39.583223 3486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance:
uuid: "5d40809d9a7e4e6c8f043333bbaaab17"
format_stamp: "Formatted at 2026-06-20 01:53:39 on dist-test-slave-wg14"
I20260620 01:53:39.594879 3486 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.009s sys 0.000s
I20260620 01:53:39.601791 3501 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:39.604581 3486 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:53:39.604889 3486 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "5d40809d9a7e4e6c8f043333bbaaab17"
format_stamp: "Formatted at 2026-06-20 01:53:39 on dist-test-slave-wg14"
I20260620 01:53:39.605195 3486 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:39.631350 3486 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:39.636966 3486 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:39.637310 3486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:39.638873 3486 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:39.641405 3486 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:39.641528 3486 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:39.641641 3486 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:39.641697 3486 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:39.692304 3486 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:40335
I20260620 01:53:39.692356 3614 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:40335 every 8 connection(s)
I20260620 01:53:39.694806 3486 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:53:39.697824 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 3486
I20260620 01:53:39.698091 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
Jun 20 01:53:39 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920419, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:53:39 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920419, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:53:39 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920419, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.131@KRBTEST.COM" created.
I20260620 01:53:39.774479 3615 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33775
I20260620 01:53:39.774382 3617 heartbeater.cc:344] Connected to a master server at 127.20.1.189:41021
I20260620 01:53:39.775054 3615 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:39.775058 3617 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:39.776665 3617 heartbeater.cc:507] Master 127.20.1.189:41021 requested a full tablet report, sending...
I20260620 01:53:39.776994 3616 heartbeater.cc:344] Connected to a master server at 127.20.1.190:38865
I20260620 01:53:39.777105 3615 heartbeater.cc:507] Master 127.20.1.188:33775 requested a full tablet report, sending...
I20260620 01:53:39.777217 3616 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:39.777962 3616 heartbeater.cc:507] Master 127.20.1.190:38865 requested a full tablet report, sending...
I20260620 01:53:39.779109 3237 ts_manager.cc:194] Registered new tserver with Master: 5d40809d9a7e4e6c8f043333bbaaab17 (127.20.1.130:40335)
I20260620 01:53:39.779177 3159 ts_manager.cc:194] Registered new tserver with Master: 5d40809d9a7e4e6c8f043333bbaaab17 (127.20.1.130:40335)
I20260620 01:53:39.780296 3237 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:51911
I20260620 01:53:39.781414 3082 ts_manager.cc:194] Registered new tserver with Master: 5d40809d9a7e4e6c8f043333bbaaab17 (127.20.1.130:40335)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:39.864431 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:53:39.992019 3472 heartbeater.cc:499] Master 127.20.1.188:33775 was elected leader, sending a full tablet report...
W20260620 01:53:40.342792 3630 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:40.343384 3630 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:40.343616 3630 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:40.354684 3630 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:40.354984 3630 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:53:40.369156 3630 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:40.372138 3630 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:40.374150 3630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:40.388228 3636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:40.388202 3635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:40.389787 3638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:40.389894 3630 server_base.cc:1061] running on GCE node
I20260620 01:53:40.390730 3630 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:40.391023 3630 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:40.391189 3630 hybrid_clock.cc:648] HybridClock initialized: now 1781920420391156 us; error 0 us; skew 500 ppm
I20260620 01:53:40.391587 3630 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:40 dist-test-slave-wg14 krb5kdc[3038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920420, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:40.396377 3630 init.cc:377] Logged in from keytab as kudu/127.20.1.131@KRBTEST.COM (short username kudu)
I20260620 01:53:40.399226 3630 webserver.cc:533] Webserver started at http://127.20.1.131:39183/ using document root <none> and password file <none>
I20260620 01:53:40.400224 3630 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:40.400393 3630 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:40.400833 3630 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:53:40.404948 3630 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/data/instance:
uuid: "f6a20000d6354872896d71dfb2d8449f"
format_stamp: "Formatted at 2026-06-20 01:53:40 on dist-test-slave-wg14"
I20260620 01:53:40.406415 3630 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance:
uuid: "f6a20000d6354872896d71dfb2d8449f"
format_stamp: "Formatted at 2026-06-20 01:53:40 on dist-test-slave-wg14"
I20260620 01:53:40.414804 3630 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 01:53:40.420631 3644 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:40.422858 3630 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20260620 01:53:40.423110 3630 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "f6a20000d6354872896d71dfb2d8449f"
format_stamp: "Formatted at 2026-06-20 01:53:40 on dist-test-slave-wg14"
I20260620 01:53:40.423386 3630 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:40.452544 3630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:40.459625 3630 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:40.460090 3630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:40.461409 3630 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:40.463837 3630 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:40.463963 3630 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:40.464075 3630 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:40.464135 3630 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:40.511718 3630 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:45803
I20260620 01:53:40.511783 3757 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:45803 every 8 connection(s)
I20260620 01:53:40.513855 3630 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:53:40.517815 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 3630
I20260620 01:53:40.518095 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
Jun 20 01:53:40 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920420, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:53:40 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920420, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:53:40 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920420, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:53:40.594900 3759 heartbeater.cc:344] Connected to a master server at 127.20.1.190:38865
I20260620 01:53:40.595512 3759 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:40.596248 3760 heartbeater.cc:344] Connected to a master server at 127.20.1.189:41021
I20260620 01:53:40.596503 3760 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:40.597047 3759 heartbeater.cc:507] Master 127.20.1.190:38865 requested a full tablet report, sending...
I20260620 01:53:40.597133 3760 heartbeater.cc:507] Master 127.20.1.189:41021 requested a full tablet report, sending...
I20260620 01:53:40.598893 3758 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33775
I20260620 01:53:40.599105 3758 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:40.599987 3758 heartbeater.cc:507] Master 127.20.1.188:33775 requested a full tablet report, sending...
I20260620 01:53:40.600425 3082 ts_manager.cc:194] Registered new tserver with Master: f6a20000d6354872896d71dfb2d8449f (127.20.1.131:45803)
I20260620 01:53:40.600550 3159 ts_manager.cc:194] Registered new tserver with Master: f6a20000d6354872896d71dfb2d8449f (127.20.1.131:45803)
I20260620 01:53:40.601759 3237 ts_manager.cc:194] Registered new tserver with Master: f6a20000d6354872896d71dfb2d8449f (127.20.1.131:45803)
I20260620 01:53:40.602937 3237 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:50097
I20260620 01:53:40.623507 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:53:40.638629 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:53:40.638854 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3052
I20260620 01:53:40.640545 3113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:40.781313 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 3052
I20260620 01:53:40.783299 3615 heartbeater.cc:499] Master 127.20.1.188:33775 was elected leader, sending a full tablet report...
W20260620 01:53:40.802985 3502 connection.cc:573] client connection to 127.20.1.190:38865 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 01:53:40.803574 3221 connection.cc:573] client connection to 127.20.1.190:38865 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 01:53:40.803858 3616 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:38865 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 01:53:40.803826 3221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:53:40.804623 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:38865
--webserver_interface=127.20.1.190
--webserver_port=36535
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:53:40.806161 3221 consensus_peers.cc:597] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af -> Peer a74f7ecbb99a46c697110c241b68fe79 (127.20.1.190:38865): Couldn't send request to peer a74f7ecbb99a46c697110c241b68fe79. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:38865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260620 01:53:40.993796 3473 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:38865 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:38865: connect: Connection refused (error 111)
I20260620 01:53:41.183651 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f007e88279d847129be498cca804b3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:41.184084 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: This master's current role is: LEADER
W20260620 01:53:41.281312 3771 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:41.281744 3771 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:41.281898 3771 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:41.291893 3771 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:41.292042 3771 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:41.292104 3771 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:41.292174 3771 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:41.306660 3771 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:38865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=36535
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:41.309741 3771 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:41.311959 3771 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:41.323931 3782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:41.323931 3783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:41.324775 3785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:41.325930 3771 server_base.cc:1061] running on GCE node
I20260620 01:53:41.326911 3771 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:41.327102 3771 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:41.327242 3771 hybrid_clock.cc:648] HybridClock initialized: now 1781920421327214 us; error 0 us; skew 500 ppm
I20260620 01:53:41.327615 3771 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:41 dist-test-slave-wg14 krb5kdc[3038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920421, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:41.332398 3771 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:53:41.335487 3771 webserver.cc:533] Webserver started at http://127.20.1.190:36535/ using document root <none> and password file <none>
I20260620 01:53:41.336535 3771 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:41.336745 3771 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:41.345285 3771 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.003s sys 0.001s
I20260620 01:53:41.350394 3791 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:41.352980 3771 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:53:41.353245 3771 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "a74f7ecbb99a46c697110c241b68fe79"
format_stamp: "Formatted at 2026-06-20 01:53:35 on dist-test-slave-wg14"
I20260620 01:53:41.354245 3771 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:41.383945 3771 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:41.390733 3771 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:41.391088 3771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:41.418779 3771 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:38865
I20260620 01:53:41.418809 3842 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:38865 every 8 connection(s)
I20260620 01:53:41.420969 3771 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:53:41.426999 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 3771
I20260620 01:53:41.427455 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3341
I20260620 01:53:41.431468 3843 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:53:41.438555 3843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: Bootstrap starting.
Jun 20 01:53:41 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 01:53:41.453917 3467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:41.465615 3843 log.cc:826] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:41.478475 3843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:53:41.479223 3843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: Bootstrap complete.
I20260620 01:53:41.487608 3843 raft_consensus.cc:359] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:41.488664 3843 raft_consensus.cc:740] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74f7ecbb99a46c697110c241b68fe79, State: Initialized, Role: FOLLOWER
I20260620 01:53:41.489696 3843 consensus_queue.cc:260] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:41.490890 3849 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:41.491353 3849 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:41.492488 3843 sys_catalog.cc:565] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:53:41.518448 3860 catalog_manager.cc:1294] Loaded cluster ID: ec16e38729174834a4f3419529641a3e
I20260620 01:53:41.518628 3860 catalog_manager.cc:1587] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: loading cluster ID for follower catalog manager: success
I20260620 01:53:41.525269 3860 catalog_manager.cc:1609] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: acquiring CA information for follower catalog manager: success
I20260620 01:53:41.529345 3860 catalog_manager.cc:1637] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Jun 20 01:53:41 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920421, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:53:41.595604 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 3341
I20260620 01:53:41.606494 3758 heartbeater.cc:499] Master 127.20.1.188:33775 was elected leader, sending a full tablet report...
I20260620 01:53:41.616801 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:36597
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--webserver_port=44371
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:53:41.621793 3808 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f6a20000d6354872896d71dfb2d8449f" instance_seqno: 1781920420499352) as {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:42315; Asking this server to re-register.
I20260620 01:53:41.623080 3759 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:41.626031 3808 ts_manager.cc:194] Registered new tserver with Master: f6a20000d6354872896d71dfb2d8449f (127.20.1.131:45803)
I20260620 01:53:41.740628 3849 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f007e88279d847129be498cca804b3af. Latest consensus state: current_term: 1 leader_uuid: "f007e88279d847129be498cca804b3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:41.741039 3849 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:41.817983 3616 heartbeater.cc:344] Connected to a master server at 127.20.1.190:38865
I20260620 01:53:41.819870 3808 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "5d40809d9a7e4e6c8f043333bbaaab17" instance_seqno: 1781920419676941) as {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:44161; Asking this server to re-register.
I20260620 01:53:41.820925 3616 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:41.823525 3808 ts_manager.cc:194] Registered new tserver with Master: 5d40809d9a7e4e6c8f043333bbaaab17 (127.20.1.130:40335)
W20260620 01:53:42.107268 3867 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:42.107686 3867 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:42.107858 3867 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:42.119837 3867 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:42.120159 3867 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:53:42.135622 3867 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:36597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=44371
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:42.138253 3867 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:42.140358 3867 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:42.155263 3874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:42.155305 3875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:42.156073 3877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:42.156870 3867 server_base.cc:1061] running on GCE node
I20260620 01:53:42.157824 3867 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:42.158026 3867 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:42.158105 3867 hybrid_clock.cc:648] HybridClock initialized: now 1781920422158085 us; error 0 us; skew 500 ppm
I20260620 01:53:42.158494 3867 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:42 dist-test-slave-wg14 krb5kdc[3038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920422, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:42.164443 3867 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:53:42.167039 3867 webserver.cc:533] Webserver started at http://127.20.1.129:44371/ using document root <none> and password file <none>
I20260620 01:53:42.168175 3867 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:42.168380 3867 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:42.177184 3867 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.003s
I20260620 01:53:42.182518 3883 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:42.184665 3867 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:53:42.184968 3867 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "024acbbf3d174b4cbe7bdee0c71b0439"
format_stamp: "Formatted at 2026-06-20 01:53:38 on dist-test-slave-wg14"
I20260620 01:53:42.186048 3867 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:42.225158 3867 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:42.236723 3867 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:42.237099 3867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:42.238451 3867 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:42.241042 3867 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:42.241338 3867 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:42.241489 3867 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:42.241567 3867 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:42.290472 3867 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:36597
I20260620 01:53:42.290577 3996 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:36597 every 8 connection(s)
I20260620 01:53:42.293695 3867 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:53:42.302382 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 3867
Jun 20 01:53:42 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920422, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:53:42 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920422, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:53:42 dist-test-slave-wg14 krb5kdc[3038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920422, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:53:42.344894 3998 heartbeater.cc:344] Connected to a master server at 127.20.1.190:38865
I20260620 01:53:42.345364 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f007e88279d847129be498cca804b3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:42.345692 3998 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:42.345746 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:42.347658 3998 heartbeater.cc:507] Master 127.20.1.190:38865 requested a full tablet report, sending...
I20260620 01:53:42.350214 3997 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33775
I20260620 01:53:42.350510 3997 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:42.351449 3997 heartbeater.cc:507] Master 127.20.1.188:33775 requested a full tablet report, sending...
I20260620 01:53:42.353098 3999 heartbeater.cc:344] Connected to a master server at 127.20.1.189:41021
I20260620 01:53:42.353358 3999 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:42.353731 3808 ts_manager.cc:194] Registered new tserver with Master: 024acbbf3d174b4cbe7bdee0c71b0439 (127.20.1.129:36597)
I20260620 01:53:42.353772 3237 ts_manager.cc:194] Re-registered known tserver with Master: 024acbbf3d174b4cbe7bdee0c71b0439 (127.20.1.129:36597)
I20260620 01:53:42.354111 3999 heartbeater.cc:507] Master 127.20.1.189:41021 requested a full tablet report, sending...
I20260620 01:53:42.355197 3237 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:55391
I20260620 01:53:42.357439 3159 ts_manager.cc:194] Re-registered known tserver with Master: 024acbbf3d174b4cbe7bdee0c71b0439 (127.20.1.129:36597)
I20260620 01:53:42.372419 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3867
W20260620 01:53:42.374426 4012 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:53:42.367951 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:34900 (local address 127.20.1.129:36597)
0620 01:53:42.368926 (+ 975us) server_negotiation.cc:207] Beginning negotiation
0620 01:53:42.368936 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0620 01:53:42.368969 (+ 33us) server_negotiation.cc:408] Connection header received
0620 01:53:42.369045 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:53:42.369053 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:53:42.369348 (+ 295us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:53:42.369591 (+ 243us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:53:42.372022 (+ 2431us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34900: BlockingRecv error: recv got EOF from 127.0.0.1:34900 (error 108)
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":559,"threads_started":1}
W20260620 01:53:42.377743 4012 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:53:42.373523 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.20.1.129:36597)
0620 01:53:42.375515 (+ 1992us) server_negotiation.cc:207] Beginning negotiation
0620 01:53:42.375523 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:53:42.375559 (+ 36us) server_negotiation.cc:408] Connection header received
0620 01:53:42.375666 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:53:42.375672 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:53:42.375776 (+ 104us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:53:42.375936 (+ 160us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:53:42.377305 (+ 1369us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34908: BlockingRecv error: recv got EOF from 127.0.0.1:34908 (error 108)
Metrics: {"server-negotiator.queue_time_us":1811,"thread_start_us":1874,"threads_started":1}
W20260620 01:53:42.377662 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3867: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:36597: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:53:42.377950 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 3867
I20260620 01:53:42.394902 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:36597
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--webserver_port=44371
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:53:42.846164 4014 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:42.846565 4014 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:42.846761 4014 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:53:42.846848 4014 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:42.857079 4014 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:42.857437 4014 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:53:42.870388 4014 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:36597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=44371
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:42.872445 4014 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:42.874161 4014 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:42.888527 4019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:42.889379 4020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:42.890281 4022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:42.891261 4014 server_base.cc:1061] running on GCE node
I20260620 01:53:42.892141 4014 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:42.892323 4014 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:42.892455 4014 hybrid_clock.cc:648] HybridClock initialized: now 1781920422892412 us; error 0 us; skew 500 ppm
I20260620 01:53:42.892817 4014 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:42 dist-test-slave-wg14 krb5kdc[3038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920422, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:42.898729 4014 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:53:42.901198 4014 webserver.cc:533] Webserver started at http://127.20.1.129:44371/ using document root <none> and password file <none>
I20260620 01:53:42.902295 4014 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:42.902431 4014 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:42.910089 4014 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:53:42.914660 4028 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:42.916776 4014 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:53:42.917054 4014 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "024acbbf3d174b4cbe7bdee0c71b0439"
format_stamp: "Formatted at 2026-06-20 01:53:38 on dist-test-slave-wg14"
I20260620 01:53:42.918076 4014 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:42.947352 4014 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:42.953962 4014 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:42.954344 4014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:42.955744 4014 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:42.958048 4014 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:42.958181 4014 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:42.958317 4014 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:42.958377 4014 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:53:42.958899 4014 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:53:42.958999 4014 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:53:42.959067 4014 fault_injection.cc:59] injecting fault for kudu (pid 4014): FLAGS_fault_before_start (process will exit)
I20260620 01:53:42.980072 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:36597
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--webserver_port=44371
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:53:43.428773 4036 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:43.429311 4036 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:43.429491 4036 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:43.439908 4036 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:43.440150 4036 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:53:43.452577 4036 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:36597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=44371
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:43.454989 4036 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:43.456931 4036 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:43.470633 4041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:43.470927 4044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:43.470760 4042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:43.472602 4036 server_base.cc:1061] running on GCE node
I20260620 01:53:43.473757 4036 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:43.474123 4036 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:43.474278 4036 hybrid_clock.cc:648] HybridClock initialized: now 1781920423474248 us; error 0 us; skew 500 ppm
I20260620 01:53:43.474882 4036 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:43 dist-test-slave-wg14 krb5kdc[3038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920423, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:43.479096 4036 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:53:43.481299 4036 webserver.cc:533] Webserver started at http://127.20.1.129:44371/ using document root <none> and password file <none>
I20260620 01:53:43.482311 4036 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:43.482468 4036 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:43.490140 4036 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.005s
I20260620 01:53:43.494362 4050 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:43.496568 4036 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:53:43.496851 4036 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "024acbbf3d174b4cbe7bdee0c71b0439"
format_stamp: "Formatted at 2026-06-20 01:53:38 on dist-test-slave-wg14"
I20260620 01:53:43.497875 4036 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:43.524088 4036 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:43.531087 4036 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:43.531548 4036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:43.532912 4036 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:43.535169 4036 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:43.535341 4036 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 01:53:43.535472 4036 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:43.535531 4036 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:43.579023 4036 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:36597
I20260620 01:53:43.579113 4163 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:36597 every 8 connection(s)
I20260620 01:53:43.581198 4036 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:53:43.591889 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 4036
I20260620 01:53:43.592406 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4036
W20260620 01:53:43.610663 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4036: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:36597: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:53:43.610913 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 4036
I20260620 01:53:43.628978 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3486
W20260620 01:53:43.629828 4169 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:53:43.586192 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:59241 (local address 127.20.1.188:33775)
0620 01:53:43.586787 (+ 595us) server_negotiation.cc:207] Beginning negotiation
0620 01:53:43.586794 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 01:53:43.600563 (+ 13769us) server_negotiation.cc:408] Connection header received
0620 01:53:43.602917 (+ 2354us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:53:43.602924 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:53:43.603084 (+ 160us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:53:43.603268 (+ 184us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:53:43.628795 (+ 25527us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:59241: BlockingRecv error: recv got EOF from 127.20.1.129:59241 (error 108)
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":1920,"threads_started":1}
W20260620 01:53:43.629899 4170 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:53:43.587513 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:59683 (local address 127.20.1.189:41021)
0620 01:53:43.604815 (+ 17302us) server_negotiation.cc:207] Beginning negotiation
0620 01:53:43.604823 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:53:43.608476 (+ 3653us) server_negotiation.cc:408] Connection header received
0620 01:53:43.609799 (+ 1323us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:53:43.609807 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:53:43.609933 (+ 126us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:53:43.610094 (+ 161us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:53:43.628891 (+ 18797us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:59683: BlockingRecv error: recv got EOF from 127.20.1.129:59683 (error 108)
Metrics: {"server-negotiator.queue_time_us":17146,"thread_start_us":17309,"threads_started":1}
W20260620 01:53:43.630685 4168 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:53:43.585845 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:46073 (local address 127.20.1.190:38865)
0620 01:53:43.587259 (+ 1414us) server_negotiation.cc:207] Beginning negotiation
0620 01:53:43.587266 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 01:53:43.604199 (+ 16933us) server_negotiation.cc:408] Connection header received
0620 01:53:43.604433 (+ 234us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:53:43.604441 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:53:43.604561 (+ 120us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:53:43.604720 (+ 159us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:53:43.611962 (+ 7242us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:53:43.613721 (+ 1759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:53:43.629710 (+ 15989us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:46073: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":1242,"thread_start_us":1098,"threads_started":1}
W20260620 01:53:43.633541 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3486: Not authorized: Client connection negotiation failed: client connection to 127.20.1.130:40335: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:53:43.633771 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 3486
I20260620 01:53:43.647802 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3630
W20260620 01:53:43.651412 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3630: Not authorized: Client connection negotiation failed: client connection to 127.20.1.131:45803: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:53:43.651618 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 3630
I20260620 01:53:43.666301 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3771
W20260620 01:53:43.669281 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3771: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:38865: server requires authentication, but client does not have Kerberos credentials available
W20260620 01:53:43.669411 4168 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:53:43.667573 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:36992 (local address 127.20.1.190:38865)
0620 01:53:43.667820 (+ 247us) server_negotiation.cc:207] Beginning negotiation
0620 01:53:43.667826 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0620 01:53:43.667852 (+ 26us) server_negotiation.cc:408] Connection header received
0620 01:53:43.668030 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:53:43.668037 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:53:43.668157 (+ 120us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:53:43.668291 (+ 134us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:53:43.669144 (+ 853us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36992: BlockingRecv error: recv got EOF from 127.0.0.1:36992 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 01:53:43.669513 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 3771
I20260620 01:53:43.695827 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3127
I20260620 01:53:43.697820 3189 generic_service.cc:196] Checking for leaks (request via RPC)
W20260620 01:53:43.777927 3221 consensus_peers.cc:597] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af -> Peer a74f7ecbb99a46c697110c241b68fe79 (127.20.1.190:38865): Couldn't send request to peer a74f7ecbb99a46c697110c241b68fe79. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:38865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 01:53:43.828753 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 3127
I20260620 01:53:43.849311 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3204
I20260620 01:53:43.851155 3267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:43.980854 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 3204
Jun 20 01:53:44 dist-test-slave-wg14 krb5kdc[4183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 01:53:44 dist-test-slave-wg14 krb5kdc[4183](info): set up 2 sockets
Jun 20 01:53:44 dist-test-slave-wg14 krb5kdc[4183](info): commencing operation
krb5kdc: starting...
W20260620 01:53:45.988677 20486 mini_kdc.cc:121] Time spent starting KDC: real 1.982s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 01:53:46 dist-test-slave-wg14 krb5kdc[4183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.190@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.190@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:46.263990 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:38865
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:53:46.719514 4197 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:46.719906 4197 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:46.720065 4197 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:46.730737 4197 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:46.730881 4197 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:46.730947 4197 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:46.730998 4197 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:46.743794 4197 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:38865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:46.745807 4197 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:46.747586 4197 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:46.758014 4203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:46.758014 4204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:46.759238 4206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:46.760365 4197 server_base.cc:1061] running on GCE node
I20260620 01:53:46.761332 4197 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:46.761535 4197 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:46.761613 4197 hybrid_clock.cc:648] HybridClock initialized: now 1781920426761588 us; error 0 us; skew 500 ppm
I20260620 01:53:46.761960 4197 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:46 dist-test-slave-wg14 krb5kdc[4183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920426, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:46.767812 4197 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:53:46.770691 4197 webserver.cc:533] Webserver started at http://127.20.1.190:42573/ using document root <none> and password file <none>
I20260620 01:53:46.771725 4197 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:46.771895 4197 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:46.781854 4197 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 01:53:46.786598 4212 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:46.789211 4197 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260620 01:53:46.789573 4197 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "a74f7ecbb99a46c697110c241b68fe79"
format_stamp: "Formatted at 2026-06-20 01:53:35 on dist-test-slave-wg14"
I20260620 01:53:46.790657 4197 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:46.827070 4197 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:46.834285 4197 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:46.834712 4197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:46.862296 4263 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:38865 every 8 connection(s)
I20260620 01:53:46.862327 4197 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:38865
I20260620 01:53:46.864666 4197 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:53:46.867923 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 4197
I20260620 01:53:46.868134 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:53:46.875105 4264 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:53:46.880473 4264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: Bootstrap starting.
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.189@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:53:46.908865 4264 log.cc:826] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:46.921679 4264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:53:46.922309 4264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: Bootstrap complete.
I20260620 01:53:46.930065 4264 raft_consensus.cc:359] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:46.931041 4264 raft_consensus.cc:740] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74f7ecbb99a46c697110c241b68fe79, State: Initialized, Role: FOLLOWER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:46.932202 4264 consensus_queue.cc:260] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:46.933401 4269 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:46.933841 4269 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:46.934741 4264 sys_catalog.cc:565] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:53:46.947870 4281 catalog_manager.cc:1294] Loaded cluster ID: ec16e38729174834a4f3419529641a3e
I20260620 01:53:46.948159 4281 catalog_manager.cc:1587] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: loading cluster ID for follower catalog manager: success
I20260620 01:53:46.953678 4281 catalog_manager.cc:1609] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: acquiring CA information for follower catalog manager: success
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.189@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:53:46.957851 4281 catalog_manager.cc:1637] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:53:46.973011 4197 master_runner.cc:187] Error getting master registration for 127.20.1.189:41021: Network error: Client connection negotiation failed: client connection to 127.20.1.189:41021: connect: Connection refused (error 111),
I20260620 01:53:46.977824 4197 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:47.000715 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:41021
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:53:47.518363 4286 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:47.518751 4286 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:47.518855 4286 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:47.529289 4286 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:47.529655 4286 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:47.529835 4286 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:47.529897 4286 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:47.544256 4286 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:41021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:47.546273 4286 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:47.548045 4286 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:47.559989 4291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:47.560034 4292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:47.561049 4294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:47.561661 4286 server_base.cc:1061] running on GCE node
I20260620 01:53:47.563401 4286 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:47.563721 4286 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:47.563867 4286 hybrid_clock.cc:648] HybridClock initialized: now 1781920427563831 us; error 0 us; skew 500 ppm
I20260620 01:53:47.564406 4286 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:47 dist-test-slave-wg14 krb5kdc[4183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920427, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:47.569828 4286 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 01:53:47.572804 4286 webserver.cc:533] Webserver started at http://127.20.1.189:42211/ using document root <none> and password file <none>
I20260620 01:53:47.573832 4286 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:47.574025 4286 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:47.581991 4286 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:53:47.587185 4300 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:47.589250 4286 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:53:47.589537 4286 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "0d2963ab3b994f0785ed4a2e74c8d616"
format_stamp: "Formatted at 2026-06-20 01:53:36 on dist-test-slave-wg14"
I20260620 01:53:47.590539 4286 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:47.619891 4286 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:47.630501 4286 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:47.630911 4286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:47.659327 4286 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:41021
I20260620 01:53:47.659364 4351 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:41021 every 8 connection(s)
I20260620 01:53:47.661576 4286 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:53:47.668537 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 4286
I20260620 01:53:47.668838 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:53:47.671805 4352 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:53:47.677080 4352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616: Bootstrap starting.
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.188@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:53:47.698875 4352 log.cc:826] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:47.712672 4352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:53:47.713867 4352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616: Bootstrap complete.
I20260620 01:53:47.722361 4352 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:47.723462 4352 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2963ab3b994f0785ed4a2e74c8d616, State: Initialized, Role: FOLLOWER
I20260620 01:53:47.725279 4352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:47.726687 4357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:47.727450 4357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: This master's current role is: FOLLOWER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:47.728523 4352 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: configured and running, proceeding with master startup.
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.188@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:53:47.759567 4369 catalog_manager.cc:1294] Loaded cluster ID: ec16e38729174834a4f3419529641a3e
I20260620 01:53:47.759804 4369 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616: loading cluster ID for follower catalog manager: success
I20260620 01:53:47.765990 4369 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616: acquiring CA information for follower catalog manager: success
Jun 20 01:53:47 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920427, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:53:47.770576 4369 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:47.799930 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:33775
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:53:47.812033 4286 master_runner.cc:187] Error getting master registration for 127.20.1.188:33775: Network error: Client connection negotiation failed: client connection to 127.20.1.188:33775: connect: Connection refused (error 111),
I20260620 01:53:47.817644 4286 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:53:48.286075 4376 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:48.286496 4376 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:48.286653 4376 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:48.298197 4376 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:48.298399 4376 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:48.298489 4376 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:48.298554 4376 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:48.311496 4376 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:33775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:48.313613 4376 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:48.315889 4376 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:48.328004 4381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:48.327975 4382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:48.328670 4384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:48.330441 4376 server_base.cc:1061] running on GCE node
I20260620 01:53:48.332316 4376 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:48.332672 4376 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:48.332800 4376 hybrid_clock.cc:648] HybridClock initialized: now 1781920428332772 us; error 0 us; skew 500 ppm
I20260620 01:53:48.333710 4376 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:48 dist-test-slave-wg14 krb5kdc[4183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920428, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:48.339905 4376 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 01:53:48.343298 4376 webserver.cc:533] Webserver started at http://127.20.1.188:35727/ using document root <none> and password file <none>
I20260620 01:53:48.344309 4376 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:48.344483 4376 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:48.353456 4376 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.008s sys 0.001s
I20260620 01:53:48.357928 4390 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:48.360579 4376 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:53:48.360834 4376 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "f007e88279d847129be498cca804b3af"
format_stamp: "Formatted at 2026-06-20 01:53:37 on dist-test-slave-wg14"
I20260620 01:53:48.361979 4376 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:48.400292 4376 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:48.406718 4376 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:48.407088 4376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:48.437636 4376 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:33775
I20260620 01:53:48.437640 4441 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:33775 every 8 connection(s)
I20260620 01:53:48.439957 4376 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:53:48.441326 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 4376
I20260620 01:53:48.441584 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:53:48.454334 4443 sys_catalog.cc:263] Verifying existing consensus state
Jun 20 01:53:48 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:53:48.462067 4443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af: Bootstrap starting.
I20260620 01:53:48.464298 4449 raft_consensus.cc:493] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:53:48.464764 4449 raft_consensus.cc:515] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:48.468192 4449 leader_election.cc:290] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d2963ab3b994f0785ed4a2e74c8d616 (127.20.1.189:41021), f007e88279d847129be498cca804b3af (127.20.1.188:33775)
Jun 20 01:53:48 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:53:48.492175 4327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a74f7ecbb99a46c697110c241b68fe79" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" is_pre_election: true
I20260620 01:53:48.492813 4327 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74f7ecbb99a46c697110c241b68fe79 in term 1.
I20260620 01:53:48.493925 4216 leader_election.cc:304] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d2963ab3b994f0785ed4a2e74c8d616, a74f7ecbb99a46c697110c241b68fe79; no voters:
I20260620 01:53:48.494643 4449 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:53:48.495012 4449 raft_consensus.cc:493] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:53:48.495205 4449 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 1 FOLLOWER]: Advancing to term 2
Jun 20 01:53:48 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920426, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:53:48.498200 4443 log.cc:826] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:48.505177 4449 raft_consensus.cc:515] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:48.507253 4327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a74f7ecbb99a46c697110c241b68fe79" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d2963ab3b994f0785ed4a2e74c8d616"
I20260620 01:53:48.507615 4327 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 1 FOLLOWER]: Advancing to term 2
Jun 20 01:53:48 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:53:48.515761 4327 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74f7ecbb99a46c697110c241b68fe79 in term 2.
I20260620 01:53:48.516611 4216 leader_election.cc:304] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d2963ab3b994f0785ed4a2e74c8d616, a74f7ecbb99a46c697110c241b68fe79; no voters:
I20260620 01:53:48.516563 4449 leader_election.cc:290] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [CANDIDATE]: Term 2 election: Requested vote from peers 0d2963ab3b994f0785ed4a2e74c8d616 (127.20.1.189:41021), f007e88279d847129be498cca804b3af (127.20.1.188:33775)
I20260620 01:53:48.517581 4449 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 01:53:48.518100 4449 raft_consensus.cc:697] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 2 LEADER]: Becoming Leader. State: Replica: a74f7ecbb99a46c697110c241b68fe79, State: Running, Role: LEADER
I20260620 01:53:48.519402 4449 consensus_queue.cc:237] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:48.522639 4443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:53:48.523828 4443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af: Bootstrap complete.
I20260620 01:53:48.530623 4455 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a74f7ecbb99a46c697110c241b68fe79. Latest consensus state: current_term: 2 leader_uuid: "a74f7ecbb99a46c697110c241b68fe79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:48.533905 4455 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:48.536382 4443 raft_consensus.cc:359] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:48.537606 4443 raft_consensus.cc:740] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f007e88279d847129be498cca804b3af, State: Initialized, Role: FOLLOWER
I20260620 01:53:48.538829 4443 consensus_queue.cc:260] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:48.540196 4457 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:53:48.540252 4458 sys_catalog.cc:455] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:48.540678 4458 sys_catalog.cc:458] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:48.541963 4443 sys_catalog.cc:565] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:53:48.541996 4417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a74f7ecbb99a46c697110c241b68fe79" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f007e88279d847129be498cca804b3af" is_pre_election: true
I20260620 01:53:48.542809 4417 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74f7ecbb99a46c697110c241b68fe79 in term 1.
I20260620 01:53:48.543361 4457 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:53:48.543766 4416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a74f7ecbb99a46c697110c241b68fe79" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f007e88279d847129be498cca804b3af"
I20260620 01:53:48.544104 4416 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:53:48.546051 4457 catalog_manager.cc:1294] Loaded cluster ID: ec16e38729174834a4f3419529641a3e
I20260620 01:53:48.546202 4457 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:53:48.549137 4457 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:53:48.551360 4457 catalog_manager.cc:6080] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: Loaded TSK: 0
I20260620 01:53:48.552177 4416 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74f7ecbb99a46c697110c241b68fe79 in term 2.
I20260620 01:53:48.553301 4457 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:53:48.569983 4469 catalog_manager.cc:1294] Loaded cluster ID: ec16e38729174834a4f3419529641a3e
I20260620 01:53:48.570152 4469 catalog_manager.cc:1587] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af: loading cluster ID for follower catalog manager: success
I20260620 01:53:48.577443 4469 catalog_manager.cc:1609] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af: acquiring CA information for follower catalog manager: success
I20260620 01:53:48.580646 4469 catalog_manager.cc:1637] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Jun 20 01:53:48 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920428, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.129@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.129@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:48.700053 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:53:49.021872 4416 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 2 FOLLOWER]: Refusing update from remote peer a74f7ecbb99a46c697110c241b68fe79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:53:49.023684 4455 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 01:53:49.031351 4458 sys_catalog.cc:455] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: SysCatalogTable state changed. Reason: New leader a74f7ecbb99a46c697110c241b68fe79. Latest consensus state: current_term: 2 leader_uuid: "a74f7ecbb99a46c697110c241b68fe79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:49.031709 4458 sys_catalog.cc:458] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:49.035432 4449 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a74f7ecbb99a46c697110c241b68fe79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:49.035779 4449 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:49.038553 4327 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 2 FOLLOWER]: Refusing update from remote peer a74f7ecbb99a46c697110c241b68fe79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:53:49.040431 4455 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 01:53:49.059356 4458 sys_catalog.cc:455] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a74f7ecbb99a46c697110c241b68fe79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:49.059684 4458 sys_catalog.cc:458] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:49.061581 4483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a74f7ecbb99a46c697110c241b68fe79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:49.061978 4483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:49.062707 4480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a74f7ecbb99a46c697110c241b68fe79. Latest consensus state: current_term: 2 leader_uuid: "a74f7ecbb99a46c697110c241b68fe79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:49.063040 4480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:53:49.182745 4476 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:49.183270 4476 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:49.183465 4476 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:49.194175 4476 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:49.194410 4476 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:53:49.211848 4476 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:49.214371 4476 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:49.216405 4476 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:49.230083 4488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:49.230482 4491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:49.230167 4489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:49.231698 4476 server_base.cc:1061] running on GCE node
I20260620 01:53:49.232524 4476 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:49.232717 4476 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:49.232822 4476 hybrid_clock.cc:648] HybridClock initialized: now 1781920429232794 us; error 0 us; skew 500 ppm
I20260620 01:53:49.233197 4476 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:49 dist-test-slave-wg14 krb5kdc[4183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920429, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:49.247989 4476 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:53:49.251295 4476 webserver.cc:533] Webserver started at http://127.20.1.129:44415/ using document root <none> and password file <none>
I20260620 01:53:49.252460 4476 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:49.252660 4476 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:49.261337 4476 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260620 01:53:49.265993 4497 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:49.268409 4476 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.004s
I20260620 01:53:49.268709 4476 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "024acbbf3d174b4cbe7bdee0c71b0439"
format_stamp: "Formatted at 2026-06-20 01:53:38 on dist-test-slave-wg14"
I20260620 01:53:49.269666 4476 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:49.292284 4476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:49.298842 4476 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:49.299266 4476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:49.300606 4476 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:49.302928 4476 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:49.303071 4476 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:49.303249 4476 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:49.303340 4476 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:49.350566 4476 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35199
I20260620 01:53:49.350660 4610 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35199 every 8 connection(s)
I20260620 01:53:49.352944 4476 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:53:49.356521 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 4476
I20260620 01:53:49.356819 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 01:53:49 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920429, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:53:49 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920429, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:53:49 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920429, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.130@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.130@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:53:49.421443 4611 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33775
I20260620 01:53:49.421545 4612 heartbeater.cc:344] Connected to a master server at 127.20.1.190:38865
I20260620 01:53:49.421623 4613 heartbeater.cc:344] Connected to a master server at 127.20.1.189:41021
I20260620 01:53:49.422098 4612 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:49.422103 4611 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:49.422097 4613 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:49.424269 4613 heartbeater.cc:507] Master 127.20.1.189:41021 requested a full tablet report, sending...
I20260620 01:53:49.424367 4611 heartbeater.cc:507] Master 127.20.1.188:33775 requested a full tablet report, sending...
I20260620 01:53:49.424392 4612 heartbeater.cc:507] Master 127.20.1.190:38865 requested a full tablet report, sending...
I20260620 01:53:49.428786 4407 ts_manager.cc:194] Registered new tserver with Master: 024acbbf3d174b4cbe7bdee0c71b0439 (127.20.1.129:35199)
I20260620 01:53:49.428808 4317 ts_manager.cc:194] Registered new tserver with Master: 024acbbf3d174b4cbe7bdee0c71b0439 (127.20.1.129:35199)
I20260620 01:53:49.430855 4229 ts_manager.cc:194] Registered new tserver with Master: 024acbbf3d174b4cbe7bdee0c71b0439 (127.20.1.129:35199)
I20260620 01:53:49.433029 4229 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:42639
I20260620 01:53:49.452172 4455 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a74f7ecbb99a46c697110c241b68fe79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:49.452550 4455 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.130 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.130 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.130@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.130@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.130 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.130 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:49.511471 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:53:49.992941 4626 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:49.993351 4626 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:49.993487 4626 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:50.005826 4626 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:50.006085 4626 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:53:50.020957 4626 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:50.022891 4626 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:50.024629 4626 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:50.039352 4634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:50.039238 4631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:50.039819 4632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:50.041306 4626 server_base.cc:1061] running on GCE node
I20260620 01:53:50.042066 4626 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:50.042269 4626 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:50.042356 4626 hybrid_clock.cc:648] HybridClock initialized: now 1781920430042331 us; error 0 us; skew 500 ppm
I20260620 01:53:50.042671 4626 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:50 dist-test-slave-wg14 krb5kdc[4183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920430, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:50.047307 4626 init.cc:377] Logged in from keytab as kudu/127.20.1.130@KRBTEST.COM (short username kudu)
I20260620 01:53:50.049774 4626 webserver.cc:533] Webserver started at http://127.20.1.130:46593/ using document root <none> and password file <none>
I20260620 01:53:50.050690 4626 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:50.050858 4626 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:50.060405 4626 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.002s sys 0.005s
I20260620 01:53:50.064869 4640 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:50.067572 4626 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260620 01:53:50.067863 4626 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "5d40809d9a7e4e6c8f043333bbaaab17"
format_stamp: "Formatted at 2026-06-20 01:53:39 on dist-test-slave-wg14"
I20260620 01:53:50.069775 4626 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:50.127558 4626 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:50.133064 4626 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:50.133388 4626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:50.134900 4626 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:50.137789 4626 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:50.138196 4626 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260620 01:53:50.138660 4626 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:50.138804 4626 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:50.187366 4626 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:39187
I20260620 01:53:50.187443 4753 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:39187 every 8 connection(s)
I20260620 01:53:50.189435 4626 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:53:50.196094 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 4626
I20260620 01:53:50.196337 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
Jun 20 01:53:50 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920430, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:53:50 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920430, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:53:50 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920430, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.131@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.131@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:53:50.242524 4754 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33775
I20260620 01:53:50.242607 4755 heartbeater.cc:344] Connected to a master server at 127.20.1.190:38865
I20260620 01:53:50.243196 4754 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:50.243188 4755 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:50.243312 4756 heartbeater.cc:344] Connected to a master server at 127.20.1.189:41021
I20260620 01:53:50.243520 4756 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:50.244923 4754 heartbeater.cc:507] Master 127.20.1.188:33775 requested a full tablet report, sending...
I20260620 01:53:50.245056 4756 heartbeater.cc:507] Master 127.20.1.189:41021 requested a full tablet report, sending...
I20260620 01:53:50.244971 4755 heartbeater.cc:507] Master 127.20.1.190:38865 requested a full tablet report, sending...
I20260620 01:53:50.247833 4229 ts_manager.cc:194] Registered new tserver with Master: 5d40809d9a7e4e6c8f043333bbaaab17 (127.20.1.130:39187)
I20260620 01:53:50.249120 4229 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:43701
I20260620 01:53:50.250376 4317 ts_manager.cc:194] Registered new tserver with Master: 5d40809d9a7e4e6c8f043333bbaaab17 (127.20.1.130:39187)
I20260620 01:53:50.251425 4407 ts_manager.cc:194] Registered new tserver with Master: 5d40809d9a7e4e6c8f043333bbaaab17 (127.20.1.130:39187)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.131 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.131 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.131@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.131@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.131 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.131 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:50.324885 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:53:50.437119 4612 heartbeater.cc:499] Master 127.20.1.190:38865 was elected leader, sending a full tablet report...
W20260620 01:53:50.786908 4768 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:50.787369 4768 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:50.787523 4768 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:50.797636 4768 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:50.797927 4768 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:53:50.812482 4768 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:50.814656 4768 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:50.816546 4768 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:50.831697 4773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:50.832010 4774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:50.831748 4776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:50.833742 4768 server_base.cc:1061] running on GCE node
I20260620 01:53:50.834615 4768 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:50.834834 4768 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:50.834942 4768 hybrid_clock.cc:648] HybridClock initialized: now 1781920430834917 us; error 0 us; skew 500 ppm
I20260620 01:53:50.835320 4768 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:50 dist-test-slave-wg14 krb5kdc[4183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920430, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:50.840001 4768 init.cc:377] Logged in from keytab as kudu/127.20.1.131@KRBTEST.COM (short username kudu)
I20260620 01:53:50.842644 4768 webserver.cc:533] Webserver started at http://127.20.1.131:42429/ using document root <none> and password file <none>
I20260620 01:53:50.843777 4768 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:50.843962 4768 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:50.852128 4768 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:53:50.857515 4782 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:50.859750 4768 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:53:50.860080 4768 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "f6a20000d6354872896d71dfb2d8449f"
format_stamp: "Formatted at 2026-06-20 01:53:40 on dist-test-slave-wg14"
I20260620 01:53:50.861267 4768 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:50.887692 4768 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:50.895224 4768 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:50.895707 4768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:50.897044 4768 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:50.899331 4768 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:50.899487 4768 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:50.899621 4768 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:50.899718 4768 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:50.945456 4768 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:41233
I20260620 01:53:50.945659 4895 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:41233 every 8 connection(s)
I20260620 01:53:50.947599 4768 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:53:50.950057 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 4768
I20260620 01:53:50.950372 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
Jun 20 01:53:50 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920430, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:53:50 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920430, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:53:50 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920430, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:53:50.996748 4896 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33775
I20260620 01:53:50.996846 4897 heartbeater.cc:344] Connected to a master server at 127.20.1.190:38865
I20260620 01:53:50.996780 4898 heartbeater.cc:344] Connected to a master server at 127.20.1.189:41021
I20260620 01:53:50.997372 4897 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:50.997447 4896 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:50.997365 4898 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:50.999161 4897 heartbeater.cc:507] Master 127.20.1.190:38865 requested a full tablet report, sending...
I20260620 01:53:50.999739 4898 heartbeater.cc:507] Master 127.20.1.189:41021 requested a full tablet report, sending...
I20260620 01:53:50.998986 4896 heartbeater.cc:507] Master 127.20.1.188:33775 requested a full tablet report, sending...
I20260620 01:53:51.002278 4407 ts_manager.cc:194] Registered new tserver with Master: f6a20000d6354872896d71dfb2d8449f (127.20.1.131:41233)
I20260620 01:53:51.003613 4229 ts_manager.cc:194] Registered new tserver with Master: f6a20000d6354872896d71dfb2d8449f (127.20.1.131:41233)
I20260620 01:53:51.004099 4317 ts_manager.cc:194] Registered new tserver with Master: f6a20000d6354872896d71dfb2d8449f (127.20.1.131:41233)
I20260620 01:53:51.005038 4229 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:56325
I20260620 01:53:51.029997 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:53:51.047220 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:53:51.047513 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4197
I20260620 01:53:51.049533 4259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:51.182780 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 4197
I20260620 01:53:51.215090 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:38865
--webserver_interface=127.20.1.190
--webserver_port=42573
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:53:51.253355 4755 heartbeater.cc:499] Master 127.20.1.190:38865 was elected leader, sending a full tablet report...
W20260620 01:53:51.255403 4755 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:38865 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:38865: connect: Connection refused (error 111)
W20260620 01:53:51.442090 4612 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:38865 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:38865: connect: Connection refused (error 111)
W20260620 01:53:51.682222 4907 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:51.682659 4907 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:51.682780 4907 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:51.693609 4907 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:51.693807 4907 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:51.693871 4907 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:51.693931 4907 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:51.709288 4907 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:38865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=42573
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:51.711555 4907 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:51.713442 4907 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:51.724565 4914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:51.725620 4915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:51.726886 4917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:51.728351 4907 server_base.cc:1061] running on GCE node
I20260620 01:53:51.729957 4907 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:51.730229 4907 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:51.730602 4907 hybrid_clock.cc:648] HybridClock initialized: now 1781920431730517 us; error 0 us; skew 500 ppm
I20260620 01:53:51.731339 4907 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:51 dist-test-slave-wg14 krb5kdc[4183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920431, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:51.736575 4907 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:53:51.740070 4907 webserver.cc:533] Webserver started at http://127.20.1.190:42573/ using document root <none> and password file <none>
I20260620 01:53:51.741094 4907 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:51.741261 4907 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:51.749333 4907 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 01:53:51.754136 4923 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:51.756585 4907 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260620 01:53:51.756841 4907 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "a74f7ecbb99a46c697110c241b68fe79"
format_stamp: "Formatted at 2026-06-20 01:53:35 on dist-test-slave-wg14"
I20260620 01:53:51.758141 4907 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:51.781574 4907 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:51.788283 4907 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:51.788687 4907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:51.817894 4975 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:38865 every 8 connection(s)
I20260620 01:53:51.817917 4907 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:38865
I20260620 01:53:51.823171 4907 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:53:51.828017 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 4907
I20260620 01:53:51.828670 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4476
I20260620 01:53:51.837709 4976 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:53:51.842527 4976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: Bootstrap starting.
Jun 20 01:53:51 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 01:53:51.861675 4606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:51.869190 4976 log.cc:826] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:51.887568 4976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:53:51.889571 4976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: Bootstrap complete.
I20260620 01:53:51.904281 4976 raft_consensus.cc:359] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:51.905730 4976 raft_consensus.cc:740] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74f7ecbb99a46c697110c241b68fe79, State: Initialized, Role: FOLLOWER
I20260620 01:53:51.907037 4976 consensus_queue.cc:260] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:51.908802 4982 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:51.909505 4982 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:51.911290 4976 sys_catalog.cc:565] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:53:51.927459 4993 catalog_manager.cc:1294] Loaded cluster ID: ec16e38729174834a4f3419529641a3e
I20260620 01:53:51.927771 4993 catalog_manager.cc:1587] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: loading cluster ID for follower catalog manager: success
I20260620 01:53:51.936107 4993 catalog_manager.cc:1609] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: acquiring CA information for follower catalog manager: success
I20260620 01:53:51.939875 4993 catalog_manager.cc:1637] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Jun 20 01:53:51 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920431, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:53:52.008757 4897 heartbeater.cc:499] Master 127.20.1.190:38865 was elected leader, sending a full tablet report...
I20260620 01:53:52.016644 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 4476
I20260620 01:53:52.032202 4907 master_runner.cc:264] No leader master found from master a74f7ecbb99a46c697110c241b68fe79
I20260620 01:53:52.032584 4907 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:53:52.037614 4940 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f6a20000d6354872896d71dfb2d8449f" instance_seqno: 1781920430933429) as {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:37741; Asking this server to re-register.
I20260620 01:53:52.039460 4897 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:52.042179 4940 ts_manager.cc:194] Registered new tserver with Master: f6a20000d6354872896d71dfb2d8449f (127.20.1.131:41233)
I20260620 01:53:52.045589 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35199
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--webserver_port=44415
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:53:52.270404 4755 heartbeater.cc:344] Connected to a master server at 127.20.1.190:38865
I20260620 01:53:52.270807 4755 heartbeater.cc:499] Master 127.20.1.190:38865 was elected leader, sending a full tablet report...
I20260620 01:53:52.272651 4940 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "5d40809d9a7e4e6c8f043333bbaaab17" instance_seqno: 1781920430176452) as {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:52993; Asking this server to re-register.
I20260620 01:53:52.273852 4755 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:52.275909 4940 ts_manager.cc:194] Registered new tserver with Master: 5d40809d9a7e4e6c8f043333bbaaab17 (127.20.1.130:39187)
W20260620 01:53:52.547398 5000 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:52.547775 5000 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:52.547917 5000 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:52.559588 5000 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:52.559834 5000 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:53:52.573688 5000 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=44415
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:52.575682 5000 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:52.577322 5000 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:52.590313 5007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:52.590883 5009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:52.591162 5006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:52.594061 5000 server_base.cc:1061] running on GCE node
I20260620 01:53:52.595705 5000 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:52.595957 5000 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:52.596055 5000 hybrid_clock.cc:648] HybridClock initialized: now 1781920432596027 us; error 0 us; skew 500 ppm
I20260620 01:53:52.596529 5000 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:52 dist-test-slave-wg14 krb5kdc[4183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920432, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:52.601827 5000 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:53:52.604377 5000 webserver.cc:533] Webserver started at http://127.20.1.129:44415/ using document root <none> and password file <none>
I20260620 01:53:52.605485 5000 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:52.605674 5000 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:52.613164 5000 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 01:53:52.618253 5015 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:52.621618 5000 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.005s sys 0.001s
I20260620 01:53:52.621897 5000 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "024acbbf3d174b4cbe7bdee0c71b0439"
format_stamp: "Formatted at 2026-06-20 01:53:38 on dist-test-slave-wg14"
I20260620 01:53:52.622949 5000 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:52.640631 5000 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:52.651528 5000 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:52.651981 5000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:52.653805 5000 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:52.657382 5000 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:52.657601 5000 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:52.657768 5000 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:52.657959 5000 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:52.687700 5072 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a74f7ecbb99a46c697110c241b68fe79)
I20260620 01:53:52.688184 5072 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:52.688395 5075 raft_consensus.cc:493] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a74f7ecbb99a46c697110c241b68fe79)
I20260620 01:53:52.688719 5075 raft_consensus.cc:515] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:52.697011 5072 leader_election.cc:290] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a74f7ecbb99a46c697110c241b68fe79 (127.20.1.190:38865), f007e88279d847129be498cca804b3af (127.20.1.188:33775)
I20260620 01:53:52.705466 5075 leader_election.cc:290] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a74f7ecbb99a46c697110c241b68fe79 (127.20.1.190:38865), 0d2963ab3b994f0785ed4a2e74c8d616 (127.20.1.189:41021)
I20260620 01:53:52.706419 4327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f007e88279d847129be498cca804b3af" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" is_pre_election: true
I20260620 01:53:52.706844 4327 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f007e88279d847129be498cca804b3af in term 2.
I20260620 01:53:52.708231 4394 leader_election.cc:304] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d2963ab3b994f0785ed4a2e74c8d616, f007e88279d847129be498cca804b3af; no voters:
I20260620 01:53:52.711535 5075 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 01:53:52.711836 5075 raft_consensus.cc:493] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 2 FOLLOWER]: Starting leader election (detected failure of leader a74f7ecbb99a46c697110c241b68fe79)
I20260620 01:53:52.711977 5075 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:53:52.715899 5075 raft_consensus.cc:515] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:52.717960 4327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f007e88279d847129be498cca804b3af" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d2963ab3b994f0785ed4a2e74c8d616"
I20260620 01:53:52.718276 4327 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:53:52.720396 4416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f007e88279d847129be498cca804b3af" is_pre_election: true
I20260620 01:53:52.720319 4950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a74f7ecbb99a46c697110c241b68fe79" is_pre_election: true
I20260620 01:53:52.720749 4416 raft_consensus.cc:2393] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d2963ab3b994f0785ed4a2e74c8d616 in current term 3: Already voted for candidate f007e88279d847129be498cca804b3af in this term.
I20260620 01:53:52.721150 4950 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d2963ab3b994f0785ed4a2e74c8d616 in term 2.
I20260620 01:53:52.721505 4327 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f007e88279d847129be498cca804b3af in term 3.
I20260620 01:53:52.722057 4394 leader_election.cc:304] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d2963ab3b994f0785ed4a2e74c8d616, f007e88279d847129be498cca804b3af; no voters:
I20260620 01:53:52.722221 5075 leader_election.cc:290] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [CANDIDATE]: Term 3 election: Requested vote from peers a74f7ecbb99a46c697110c241b68fe79 (127.20.1.190:38865), 0d2963ab3b994f0785ed4a2e74c8d616 (127.20.1.189:41021)
I20260620 01:53:52.722483 5075 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 3 FOLLOWER]: Leader election won for term 3
I20260620 01:53:52.722342 4301 leader_election.cc:304] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d2963ab3b994f0785ed4a2e74c8d616, a74f7ecbb99a46c697110c241b68fe79; no voters: f007e88279d847129be498cca804b3af
I20260620 01:53:52.722682 5075 raft_consensus.cc:697] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [term 3 LEADER]: Becoming Leader. State: Replica: f007e88279d847129be498cca804b3af, State: Running, Role: LEADER
I20260620 01:53:52.722954 5072 raft_consensus.cc:2764] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260620 01:53:52.724210 5075 consensus_queue.cc:237] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } }
I20260620 01:53:52.728372 5075 sys_catalog.cc:455] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: SysCatalogTable state changed. Reason: New leader f007e88279d847129be498cca804b3af. Latest consensus state: current_term: 3 leader_uuid: "f007e88279d847129be498cca804b3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:52.728935 5075 sys_catalog.cc:458] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:52.735309 5091 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:53:52.736832 4950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f007e88279d847129be498cca804b3af" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a74f7ecbb99a46c697110c241b68fe79" is_pre_election: true
I20260620 01:53:52.736970 4949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f007e88279d847129be498cca804b3af" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a74f7ecbb99a46c697110c241b68fe79"
I20260620 01:53:52.737223 4950 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f007e88279d847129be498cca804b3af in term 2.
I20260620 01:53:52.738773 5091 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:53:52.740381 5091 catalog_manager.cc:1294] Loaded cluster ID: ec16e38729174834a4f3419529641a3e
I20260620 01:53:52.740533 5091 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:53:52.742520 5091 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:53:52.744216 5091 catalog_manager.cc:6080] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af: Loaded TSK: 0
I20260620 01:53:52.745718 5091 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:53:52.754532 5000 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35199
I20260620 01:53:52.754707 5137 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35199 every 8 connection(s)
I20260620 01:53:52.756896 5000 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:53:52.763648 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 5000
Jun 20 01:53:52 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920432, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:53:52 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920432, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:53:52 dist-test-slave-wg14 krb5kdc[4183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920432, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:53:52.800371 5140 heartbeater.cc:344] Connected to a master server at 127.20.1.189:41021
I20260620 01:53:52.800938 5140 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:52.802260 5140 heartbeater.cc:507] Master 127.20.1.189:41021 requested a full tablet report, sending...
I20260620 01:53:52.803296 5139 heartbeater.cc:344] Connected to a master server at 127.20.1.190:38865
I20260620 01:53:52.803511 5139 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:52.804201 5139 heartbeater.cc:507] Master 127.20.1.190:38865 requested a full tablet report, sending...
I20260620 01:53:52.804776 5138 heartbeater.cc:344] Connected to a master server at 127.20.1.188:33775
I20260620 01:53:52.804949 5138 heartbeater.cc:461] Registering TS with master...
I20260620 01:53:52.805234 4317 ts_manager.cc:194] Re-registered known tserver with Master: 024acbbf3d174b4cbe7bdee0c71b0439 (127.20.1.129:35199)
I20260620 01:53:52.805568 5138 heartbeater.cc:507] Master 127.20.1.188:33775 requested a full tablet report, sending...
I20260620 01:53:52.805900 4940 ts_manager.cc:194] Registered new tserver with Master: 024acbbf3d174b4cbe7bdee0c71b0439 (127.20.1.129:35199)
I20260620 01:53:52.807184 4407 ts_manager.cc:194] Re-registered known tserver with Master: 024acbbf3d174b4cbe7bdee0c71b0439 (127.20.1.129:35199)
I20260620 01:53:52.809484 4407 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:35077
I20260620 01:53:52.821571 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 5000
W20260620 01:53:52.822508 5149 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:53:52.816003 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.20.1.129:35199)
0620 01:53:52.817782 (+ 1779us) server_negotiation.cc:207] Beginning negotiation
0620 01:53:52.817792 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0620 01:53:52.817841 (+ 49us) server_negotiation.cc:408] Connection header received
0620 01:53:52.817955 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:53:52.817967 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:53:52.818568 (+ 601us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:53:52.818832 (+ 264us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:53:52.821019 (+ 2187us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39886: BlockingRecv error: recv got EOF from 127.0.0.1:39886 (error 108)
Metrics: {"server-negotiator.queue_time_us":1536,"thread_start_us":1325,"threads_started":1}
W20260620 01:53:52.826997 5149 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:53:52.822900 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.20.1.129:35199)
0620 01:53:52.823276 (+ 376us) server_negotiation.cc:207] Beginning negotiation
0620 01:53:52.823283 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 01:53:52.825373 (+ 2090us) server_negotiation.cc:408] Connection header received
0620 01:53:52.825610 (+ 237us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:53:52.825618 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:53:52.825727 (+ 109us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:53:52.825854 (+ 127us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:53:52.826715 (+ 861us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39890: BlockingRecv error: recv got EOF from 127.0.0.1:39890 (error 108)
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":1973,"threads_started":1}
W20260620 01:53:52.827685 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 5000: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:35199: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:53:52.827924 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 5000
I20260620 01:53:52.829857 4286 master_runner.cc:233] Existing masters have differing terms: 2 vs 3
I20260620 01:53:52.830559 4286 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:53:52.846202 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35199
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--webserver_port=44415
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:53:53.135622 4949 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:53:53.141260 4949 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [term 3 FOLLOWER]: Refusing update from remote peer f007e88279d847129be498cca804b3af: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:53:53.142830 5075 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:53:53.148940 5152 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f007e88279d847129be498cca804b3af. Latest consensus state: current_term: 3 leader_uuid: "f007e88279d847129be498cca804b3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:53.149523 5152 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:53.152726 5088 sys_catalog.cc:455] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f007e88279d847129be498cca804b3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:53.153400 5088 sys_catalog.cc:458] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:53.155856 4327 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [term 3 FOLLOWER]: Refusing update from remote peer f007e88279d847129be498cca804b3af: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:53:53.157390 5088 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:53:53.167909 5152 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f007e88279d847129be498cca804b3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:53.168224 5152 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74f7ecbb99a46c697110c241b68fe79 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:53.172048 5159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f007e88279d847129be498cca804b3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:53.172488 5159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:53.172039 5072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f007e88279d847129be498cca804b3af. Latest consensus state: current_term: 3 leader_uuid: "f007e88279d847129be498cca804b3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:53.172683 5072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2963ab3b994f0785ed4a2e74c8d616 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:53:53.313874 5151 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:53.314304 5151 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:53.314464 5151 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:53:53.314543 5151 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:53.325659 5151 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:53.326000 5151 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:53:53.341181 5151 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=44415
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:53.343880 5151 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:53.346087 5151 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:53.361518 5165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:53.361930 5167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:53.362742 5169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:53.363523 5151 server_base.cc:1061] running on GCE node
I20260620 01:53:53.364379 5151 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:53.364566 5151 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:53.364707 5151 hybrid_clock.cc:648] HybridClock initialized: now 1781920433364658 us; error 0 us; skew 500 ppm
I20260620 01:53:53.365090 5151 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:53 dist-test-slave-wg14 krb5kdc[4183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920433, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:53.370766 5151 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:53:53.374254 5151 webserver.cc:533] Webserver started at http://127.20.1.129:44415/ using document root <none> and password file <none>
I20260620 01:53:53.376148 5151 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:53.376329 5151 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:53.384452 5151 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 01:53:53.389372 5175 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:53.392067 5151 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260620 01:53:53.392340 5151 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "024acbbf3d174b4cbe7bdee0c71b0439"
format_stamp: "Formatted at 2026-06-20 01:53:38 on dist-test-slave-wg14"
I20260620 01:53:53.393839 5151 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:53.422868 5151 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:53.428789 5151 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:53.429183 5151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:53.430481 5151 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:53.433495 5151 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:53.433717 5151 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:53.433882 5151 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:53.433936 5151 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:53:53.434526 5151 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:53:53.434638 5151 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:53:53.434706 5151 fault_injection.cc:59] injecting fault for kudu (pid 5151): FLAGS_fault_before_start (process will exit)
I20260620 01:53:53.460839 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35199
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--webserver_port=44415
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:53:53.641328 5155 sys_catalog.cc:455] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f007e88279d847129be498cca804b3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f7ecbb99a46c697110c241b68fe79" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 38865 } } peers { permanent_uuid: "0d2963ab3b994f0785ed4a2e74c8d616" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 41021 } } peers { permanent_uuid: "f007e88279d847129be498cca804b3af" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 33775 } } }
I20260620 01:53:53.641686 5155 sys_catalog.cc:458] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af [sys.catalog]: This master's current role is: LEADER
W20260620 01:53:53.930868 5183 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:53.931382 5183 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:53.931553 5183 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:53.943871 5183 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:53.944136 5183 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:53:53.960664 5183 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=44415
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:38865,127.20.1.189:41021,127.20.1.188:33775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:53.962808 5183 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:53.964924 5183 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:53.981729 5189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:53.981776 5188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:53.987594 5191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:53.987912 5183 server_base.cc:1061] running on GCE node
I20260620 01:53:53.989157 5183 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:53:53.989411 5183 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:53:53.989518 5183 hybrid_clock.cc:648] HybridClock initialized: now 1781920433989490 us; error 0 us; skew 500 ppm
I20260620 01:53:53.989970 5183 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:53 dist-test-slave-wg14 krb5kdc[4183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920433, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:53.996491 5183 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:53:54.001014 5183 webserver.cc:533] Webserver started at http://127.20.1.129:44415/ using document root <none> and password file <none>
I20260620 01:53:54.002040 5183 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:54.002213 5183 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:54.010850 5183 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 01:53:54.011557 4896 heartbeater.cc:499] Master 127.20.1.188:33775 was elected leader, sending a full tablet report...
I20260620 01:53:54.016834 5198 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:54.019470 5183 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:53:54.019730 5183 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "024acbbf3d174b4cbe7bdee0c71b0439"
format_stamp: "Formatted at 2026-06-20 01:53:38 on dist-test-slave-wg14"
I20260620 01:53:54.020712 5183 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:54.054942 5183 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:54.064983 5183 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:54.065851 5183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:54.067837 5183 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:53:54.070931 5183 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:53:54.071171 5183 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:54.071298 5183 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:53:54.071372 5183 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:54.128436 5183 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35199
I20260620 01:53:54.128537 5311 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35199 every 8 connection(s)
I20260620 01:53:54.131896 5183 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:53:54.140223 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 5183
I20260620 01:53:54.140585 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 5183
W20260620 01:53:54.168174 5323 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:53:54.142578 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:39906 (local address 127.20.1.129:35199)
0620 01:53:54.153106 (+ 10528us) server_negotiation.cc:207] Beginning negotiation
0620 01:53:54.153141 (+ 35us) server_negotiation.cc:400] Waiting for connection header
0620 01:53:54.161669 (+ 8528us) server_negotiation.cc:408] Connection header received
0620 01:53:54.162064 (+ 395us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:53:54.162074 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:53:54.162442 (+ 368us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:53:54.163037 (+ 595us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:53:54.165450 (+ 2413us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39906: BlockingRecv error: recv got EOF from 127.0.0.1:39906 (error 108)
Metrics: {"server-negotiator.queue_time_us":10142,"thread_start_us":10023,"threads_started":1}
W20260620 01:53:54.170178 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 5183: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:35199: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:53:54.170490 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 5183
I20260620 01:53:54.188877 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4626
W20260620 01:53:54.190115 5318 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:53:54.137644 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:44557 (local address 127.20.1.190:38865)
0620 01:53:54.155364 (+ 17720us) server_negotiation.cc:207] Beginning negotiation
0620 01:53:54.155372 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:53:54.159946 (+ 4574us) server_negotiation.cc:408] Connection header received
0620 01:53:54.160337 (+ 391us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:53:54.160347 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:53:54.160463 (+ 116us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:53:54.160591 (+ 128us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:53:54.163676 (+ 3085us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:53:54.165323 (+ 1647us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:53:54.168755 (+ 3432us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:53:54.170014 (+ 1259us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0620 01:53:54.188763 (+ 18749us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:44557: BlockingRecv error: recv got EOF from 127.20.1.129:44557 (error 108)
Metrics: {"server-negotiator.queue_time_us":17568,"thread_start_us":17421,"threads_started":1}
W20260620 01:53:54.191946 5316 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:53:54.137326 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:60765 (local address 127.20.1.189:41021)
0620 01:53:54.152077 (+ 14751us) server_negotiation.cc:207] Beginning negotiation
0620 01:53:54.152086 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0620 01:53:54.154853 (+ 2767us) server_negotiation.cc:408] Connection header received
0620 01:53:54.157669 (+ 2816us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:53:54.157677 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:53:54.157795 (+ 118us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:53:54.157925 (+ 130us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:53:54.162847 (+ 4922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:53:54.164568 (+ 1721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:53:54.169278 (+ 4710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:53:54.170143 (+ 865us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0620 01:53:54.190811 (+ 20668us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:60765: BlockingRecv error: recv got EOF from 127.20.1.129:60765 (error 108)
Metrics: {"server-negotiator.queue_time_us":14541,"thread_start_us":14724,"threads_started":1}
W20260620 01:53:54.191923 5317 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:53:54.137470 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:44181 (local address 127.20.1.188:33775)
0620 01:53:54.153750 (+ 16280us) server_negotiation.cc:207] Beginning negotiation
0620 01:53:54.153758 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:53:54.160384 (+ 6626us) server_negotiation.cc:408] Connection header received
0620 01:53:54.161260 (+ 876us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:53:54.161267 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:53:54.161366 (+ 99us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:53:54.161510 (+ 144us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:53:54.166421 (+ 4911us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:53:54.168397 (+ 1976us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:53:54.188737 (+ 20340us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:44181: BlockingRecv error: recv got EOF from 127.20.1.129:44181 (error 108)
Metrics: {"server-negotiator.queue_time_us":16097,"thread_start_us":15951,"threads_started":1}
W20260620 01:53:54.195048 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4626: Not authorized: Client connection negotiation failed: client connection to 127.20.1.130:39187: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:53:54.195326 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 4626
I20260620 01:53:54.215955 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4768
W20260620 01:53:54.219864 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4768: Not authorized: Client connection negotiation failed: client connection to 127.20.1.131:41233: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:53:54.220196 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 4768
I20260620 01:53:54.238794 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4907
W20260620 01:53:54.242316 5318 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:53:54.240380 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.20.1.190:38865)
0620 01:53:54.240640 (+ 260us) server_negotiation.cc:207] Beginning negotiation
0620 01:53:54.240648 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:53:54.240671 (+ 23us) server_negotiation.cc:408] Connection header received
0620 01:53:54.240755 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:53:54.240761 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:53:54.240848 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:53:54.240975 (+ 127us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:53:54.241995 (+ 1020us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43254: BlockingRecv error: recv got EOF from 127.0.0.1:43254 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260620 01:53:54.242208 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4907: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:38865: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:53:54.242547 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 4907
W20260620 01:53:54.266255 4391 connection.cc:573] client connection to 127.20.1.190:38865 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:53:54.266713 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4286
W20260620 01:53:54.267096 4391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:53:54.268508 4347 generic_service.cc:196] Checking for leaks (request via RPC)
W20260620 01:53:54.270335 4391 consensus_peers.cc:597] T 00000000000000000000000000000000 P f007e88279d847129be498cca804b3af -> Peer a74f7ecbb99a46c697110c241b68fe79 (127.20.1.190:38865): Couldn't send request to peer a74f7ecbb99a46c697110c241b68fe79. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:38865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 01:53:54.415388 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 4286
I20260620 01:53:54.439728 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4376
I20260620 01:53:54.441519 4437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:53:54.581420 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 4376
[ OK ] ExternalMiniClusterTest.TestBasicOperation/6 (21793 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/7
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 01:53:54 dist-test-slave-wg14 krb5kdc[5332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 01:53:54 dist-test-slave-wg14 krb5kdc[5332](info): set up 2 sockets
Jun 20 01:53:54 dist-test-slave-wg14 krb5kdc[5332](info): commencing operation
krb5kdc: starting...
W20260620 01:53:56.664503 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.042s user 0.009s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 01:53:56 dist-test-slave-wg14 krb5kdc[5332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T01:53:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:53:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:57.026309 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44547
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:53:57.520331 5349 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:57.520753 5349 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:57.520852 5349 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:57.531450 5349 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:57.531607 5349 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:57.531682 5349 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:57.531744 5349 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:57.546427 5349 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:57.548799 5349 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:57.550807 5349 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:57.561316 5355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:57.561292 5354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:57.562379 5357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:57.563617 5349 server_base.cc:1061] running on GCE node
I20260620 01:53:57.565157 5349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:57.567629 5349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:57.569049 5349 hybrid_clock.cc:648] HybridClock initialized: now 1781920437568943 us; error 105 us; skew 500 ppm
I20260620 01:53:57.569717 5349 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:57 dist-test-slave-wg14 krb5kdc[5332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920437, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:57.574208 5349 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:53:57.576990 5349 webserver.cc:533] Webserver started at http://127.20.1.190:33709/ using document root <none> and password file <none>
I20260620 01:53:57.577975 5349 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:57.578117 5349 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:57.578545 5349 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:53:57.581858 5349 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "4345ecdc1b104a6096e8744d2614ec04"
format_stamp: "Formatted at 2026-06-20 01:53:57 on dist-test-slave-wg14"
I20260620 01:53:57.582996 5349 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "4345ecdc1b104a6096e8744d2614ec04"
format_stamp: "Formatted at 2026-06-20 01:53:57 on dist-test-slave-wg14"
I20260620 01:53:57.593241 5349 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.006s sys 0.001s
I20260620 01:53:57.598723 5364 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:57.602075 5349 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260620 01:53:57.602501 5349 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "4345ecdc1b104a6096e8744d2614ec04"
format_stamp: "Formatted at 2026-06-20 01:53:57 on dist-test-slave-wg14"
I20260620 01:53:57.603611 5349 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:57.623235 5349 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:57.630254 5349 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:57.630690 5349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:57.660889 5349 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44547
I20260620 01:53:57.661042 5415 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44547 every 8 connection(s)
I20260620 01:53:57.662858 5349 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:53:57.669726 5416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:53:57.671633 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 5349
I20260620 01:53:57.672098 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:53:57.677615 5416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.189@KRBTEST.COM" created.
Jun 20 01:53:57 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920437, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:53:57.717586 5416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:53:57.719992 5367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:34883: connect: Connection refused (error 111)
W20260620 01:53:57.722645 5416 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:34883: Network error: Client connection negotiation failed: client connection to 127.20.1.189:34883: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.189@KRBTEST.COM" created.
I20260620 01:53:57.772315 5416 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } attempt: 1
W20260620 01:53:57.774973 5416 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:34883: Network error: Client connection negotiation failed: client connection to 127.20.1.189:34883: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:57.791477 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:34883
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:53:57.843375 5416 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } attempt: 2
W20260620 01:53:57.846552 5416 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:34883: Network error: Client connection negotiation failed: client connection to 127.20.1.189:34883: connect: Connection refused (error 111)
I20260620 01:53:57.938007 5416 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } attempt: 3
W20260620 01:53:57.940925 5416 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:34883: Network error: Client connection negotiation failed: client connection to 127.20.1.189:34883: connect: Connection refused (error 111)
I20260620 01:53:58.084513 5416 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } attempt: 4
W20260620 01:53:58.087893 5416 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:34883: Network error: Client connection negotiation failed: client connection to 127.20.1.189:34883: connect: Connection refused (error 111)
W20260620 01:53:58.263360 5425 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:58.263758 5425 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:58.263875 5425 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:58.274387 5425 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:58.274565 5425 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:58.274632 5425 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:58.274675 5425 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:58.288944 5425 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:34883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:58.290879 5425 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:58.292541 5425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:58.304960 5431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:58.304949 5430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:58.306008 5433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:58.308398 5425 server_base.cc:1061] running on GCE node
I20260620 01:53:58.309533 5425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:58.311797 5425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:58.313529 5425 hybrid_clock.cc:648] HybridClock initialized: now 1781920438313411 us; error 90 us; skew 500 ppm
I20260620 01:53:58.314217 5425 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:58 dist-test-slave-wg14 krb5kdc[5332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920438, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:58.319672 5425 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 01:53:58.323557 5425 webserver.cc:533] Webserver started at http://127.20.1.189:36505/ using document root <none> and password file <none>
I20260620 01:53:58.325359 5425 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:58.325644 5425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:58.326195 5425 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:53:58.329391 5425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "cd052da5773244dd80b5fbbd78139b44"
format_stamp: "Formatted at 2026-06-20 01:53:58 on dist-test-slave-wg14"
I20260620 01:53:58.330986 5425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "cd052da5773244dd80b5fbbd78139b44"
format_stamp: "Formatted at 2026-06-20 01:53:58 on dist-test-slave-wg14"
I20260620 01:53:58.339359 5425 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260620 01:53:58.345006 5440 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:58.347600 5425 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20260620 01:53:58.347892 5425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "cd052da5773244dd80b5fbbd78139b44"
format_stamp: "Formatted at 2026-06-20 01:53:58 on dist-test-slave-wg14"
I20260620 01:53:58.348173 5425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:58.374109 5425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:58.380003 5425 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:58.380374 5425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:58.387362 5416 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } attempt: 5
W20260620 01:53:58.390460 5416 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:34883: Network error: Client connection negotiation failed: client connection to 127.20.1.189:34883: connect: Connection refused (error 111)
I20260620 01:53:58.410501 5425 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:34883
I20260620 01:53:58.410624 5492 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:34883 every 8 connection(s)
I20260620 01:53:58.413203 5425 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:53:58.417876 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 5425
I20260620 01:53:58.418145 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:53:58.421129 5493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:53:58.430646 5493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.188@KRBTEST.COM" created.
Jun 20 01:53:58 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920438, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:53:58.463006 5493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:53:58 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920438, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:58.492182 5493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:53:58.494189 5443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44059: connect: Connection refused (error 111)
W20260620 01:53:58.496555 5493 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44059: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44059: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:53:58.546104 5493 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } attempt: 1
I20260620 01:53:58.548174 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:44059
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:53:58.548839 5493 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44059: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44059: connect: Connection refused (error 111)
I20260620 01:53:58.617275 5493 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } attempt: 2
W20260620 01:53:58.620946 5493 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44059: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44059: connect: Connection refused (error 111)
I20260620 01:53:58.712301 5493 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } attempt: 3
W20260620 01:53:58.715636 5493 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44059: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44059: connect: Connection refused (error 111)
I20260620 01:53:58.859073 5493 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } attempt: 4
W20260620 01:53:58.862360 5493 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44059: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44059: connect: Connection refused (error 111)
I20260620 01:53:58.937904 5416 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } attempt: 6
Jun 20 01:53:58 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920437, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:53:58.958254 5416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:53:58.962019 5416 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44059: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44059: connect: Connection refused (error 111)
I20260620 01:53:59.014392 5416 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } attempt: 1
W20260620 01:53:59.017972 5416 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44059: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44059: connect: Connection refused (error 111)
W20260620 01:53:59.018961 5503 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:53:59.020030 5503 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:53:59.020208 5503 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:53:59.034963 5503 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:53:59.035192 5503 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:53:59.035284 5503 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:53:59.035349 5503 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:53:59.050621 5503 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:53:59.052682 5503 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:53:59.054453 5503 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:53:59.066313 5510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:59.066251 5511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:53:59.067409 5513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:53:59.068069 5503 server_base.cc:1061] running on GCE node
I20260620 01:53:59.069145 5503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:53:59.071290 5503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:53:59.072717 5503 hybrid_clock.cc:648] HybridClock initialized: now 1781920439072618 us; error 103 us; skew 500 ppm
I20260620 01:53:59.073211 5503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:53:59 dist-test-slave-wg14 krb5kdc[5332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920439, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:53:59.078574 5503 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 01:53:59.080992 5503 webserver.cc:533] Webserver started at http://127.20.1.188:43949/ using document root <none> and password file <none>
I20260620 01:53:59.082175 5503 fs_manager.cc:362] Metadata directory not provided
I20260620 01:53:59.082401 5503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:53:59.082937 5503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:53:59.086028 5503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "9058f04895554d8585c606f129db9310"
format_stamp: "Formatted at 2026-06-20 01:53:59 on dist-test-slave-wg14"
I20260620 01:53:59.087096 5503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "9058f04895554d8585c606f129db9310"
format_stamp: "Formatted at 2026-06-20 01:53:59 on dist-test-slave-wg14"
I20260620 01:53:59.092305 5416 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } attempt: 2
W20260620 01:53:59.095716 5416 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44059: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44059: connect: Connection refused (error 111)
I20260620 01:53:59.096576 5503 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.002s sys 0.005s
I20260620 01:53:59.102280 5520 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:53:59.104730 5503 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:53:59.104991 5503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "9058f04895554d8585c606f129db9310"
format_stamp: "Formatted at 2026-06-20 01:53:59 on dist-test-slave-wg14"
I20260620 01:53:59.105242 5503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:53:59.139299 5503 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:53:59.144791 5503 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:53:59.145323 5503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:53:59.161687 5493 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } attempt: 5
W20260620 01:53:59.164989 5493 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44059: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44059: connect: Connection refused (error 111)
I20260620 01:53:59.173045 5503 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:44059
I20260620 01:53:59.173063 5571 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:44059 every 8 connection(s)
I20260620 01:53:59.176108 5503 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:53:59.184551 5572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:53:59.186105 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 5503
I20260620 01:53:59.186374 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:53:59.194264 5572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:53:59 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:53:59.209031 5416 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } attempt: 3
Jun 20 01:53:59 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920439, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:53:59 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920437, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:53:59 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:53:59.232208 5572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:53:59.249295 5416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: Bootstrap starting.
Jun 20 01:53:59 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920439, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:53:59 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:53:59.255728 5416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: Neither blocks nor log segments found. Creating new log.
I20260620 01:53:59.257344 5416 log.cc:826] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:59.260850 5572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:53:59.262454 5416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: No bootstrap required, opened a new log
Jun 20 01:53:59 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920439, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:53:59.271911 5416 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:53:59.272356 5416 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:53:59.272539 5416 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4345ecdc1b104a6096e8744d2614ec04, State: Initialized, Role: FOLLOWER
I20260620 01:53:59.273402 5416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:53:59.274822 5582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:53:59.275494 5582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:59.276763 5416 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:53:59.284965 5572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310: Bootstrap starting.
I20260620 01:53:59.289678 5572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310: Neither blocks nor log segments found. Creating new log.
I20260620 01:53:59.292450 5572 log.cc:826] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310: Log is configured to *not* fsync() on all Append() calls
W20260620 01:53:59.298152 5593 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:53:59.298388 5593 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:53:59.294277 5349 master_runner.cc:187] Error getting master registration for 127.20.1.189:34883: OK, instance_id { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" instance_seqno: 1781920438402039 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:53:59.300745 5349 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:53:59.301394 5572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310: No bootstrap required, opened a new log
I20260620 01:53:59.310499 5572 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:53:59.311088 5572 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:53:59.311408 5572 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9058f04895554d8585c606f129db9310, State: Initialized, Role: FOLLOWER
I20260620 01:53:59.312427 5572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:53:59.315322 5596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:53:59.315701 5596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:59.317637 5572 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:53:59.335361 5607 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:53:59.335646 5607 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:53:59.336359 5503 master_runner.cc:187] Error getting master registration for 127.20.1.189:34883: OK, instance_id { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" instance_seqno: 1781920438402039 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:53:59.345283 5503 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:53:59.385982 5370 tablet.cc:2406] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 01:53:59.398726 5526 tablet.cc:2406] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:53:59.572744 5596 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:53:59.573494 5596 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:53:59.576258 5596 leader_election.cc:290] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4345ecdc1b104a6096e8744d2614ec04 (127.20.1.190:44547), cd052da5773244dd80b5fbbd78139b44 (127.20.1.189:34883)
I20260620 01:53:59.577986 5391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9058f04895554d8585c606f129db9310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4345ecdc1b104a6096e8744d2614ec04" is_pre_election: true
I20260620 01:53:59.578814 5391 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9058f04895554d8585c606f129db9310 in term 0.
I20260620 01:53:59.580331 5524 leader_election.cc:304] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4345ecdc1b104a6096e8744d2614ec04, 9058f04895554d8585c606f129db9310; no voters:
I20260620 01:53:59.581197 5596 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:53:59.581658 5596 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:53:59.581879 5596 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:53:59.585855 5596 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:53:59.587785 5391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9058f04895554d8585c606f129db9310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4345ecdc1b104a6096e8744d2614ec04"
I20260620 01:53:59.588142 5391 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:53:59.593420 5391 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9058f04895554d8585c606f129db9310 in term 1.
I20260620 01:53:59.594991 5524 leader_election.cc:304] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4345ecdc1b104a6096e8744d2614ec04, 9058f04895554d8585c606f129db9310; no voters:
I20260620 01:53:59.596288 5609 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:53:59.598656 5596 leader_election.cc:290] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [CANDIDATE]: Term 1 election: Requested vote from peers 4345ecdc1b104a6096e8744d2614ec04 (127.20.1.190:44547), cd052da5773244dd80b5fbbd78139b44 (127.20.1.189:34883)
I20260620 01:53:59.600049 5609 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 1 LEADER]: Becoming Leader. State: Replica: 9058f04895554d8585c606f129db9310, State: Running, Role: LEADER
I20260620 01:53:59.602465 5609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:53:59.626516 5596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9058f04895554d8585c606f129db9310. Latest consensus state: current_term: 1 leader_uuid: "9058f04895554d8585c606f129db9310" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:53:59.626931 5596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:59.628636 5612 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:53:59.632543 5612 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:53:59.645277 5391 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 1 FOLLOWER]: Refusing update from remote peer 9058f04895554d8585c606f129db9310: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:53:59.646783 5596 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:53:59.683022 5582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9058f04895554d8585c606f129db9310. Latest consensus state: current_term: 1 leader_uuid: "9058f04895554d8585c606f129db9310" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:53:59.683430 5582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:59.691819 5613 mvcc.cc:204] Tried to move back new op lower bound from 7298746120770031616 to 7298746120639373312. Current Snapshot: MvccSnapshot[applied={T|T < 7298746120770031616}]
I20260620 01:53:59.698221 5609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9058f04895554d8585c606f129db9310" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:53:59.698974 5609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:59.703712 5612 catalog_manager.cc:1382] Generated new cluster ID: 406167b0b78a4a00b6eb7f76168e5158
I20260620 01:53:59.699424 5614 mvcc.cc:204] Tried to move back new op lower bound from 7298746120770031616 to 7298746120639373312. Current Snapshot: MvccSnapshot[applied={T|T < 7298746120770031616}]
I20260620 01:53:59.703930 5612 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:53:59.712283 5493 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } attempt: 6
I20260620 01:53:59.720088 5582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9058f04895554d8585c606f129db9310" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:53:59.720436 5582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 01:53:59 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920438, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:53:59.753027 5493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44: Bootstrap starting.
I20260620 01:53:59.759647 5493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44: Neither blocks nor log segments found. Creating new log.
I20260620 01:53:59.777498 5493 log.cc:826] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44: Log is configured to *not* fsync() on all Append() calls
I20260620 01:53:59.800570 5493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44: No bootstrap required, opened a new log
I20260620 01:53:59.810890 5493 raft_consensus.cc:359] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:53:59.825495 5493 raft_consensus.cc:385] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:53:59.825728 5493 raft_consensus.cc:740] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd052da5773244dd80b5fbbd78139b44, State: Initialized, Role: FOLLOWER
I20260620 01:53:59.826993 5493 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:53:59.829151 5620 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:53:59.830433 5493 sys_catalog.cc:565] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:53:59.830915 5465 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:53:59.835623 5466 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9058f04895554d8585c606f129db9310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd052da5773244dd80b5fbbd78139b44"
I20260620 01:53:59.837225 5467 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9058f04895554d8585c606f129db9310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd052da5773244dd80b5fbbd78139b44" is_pre_election: true
I20260620 01:53:59.839511 5612 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:53:59.840058 5465 raft_consensus.cc:1275] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 1 FOLLOWER]: Refusing update from remote peer 9058f04895554d8585c606f129db9310: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:53:59.840857 5620 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:59.841730 5612 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:53:59.849426 5596 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:53:59.877882 5612 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310: Generated new TSK 0
W20260620 01:53:59.885030 5446 tablet.cc:2406] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:53:59.889792 5612 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:53:59.938624 5620 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9058f04895554d8585c606f129db9310. Latest consensus state: current_term: 1 leader_uuid: "9058f04895554d8585c606f129db9310" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:53:59.939047 5620 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:59.938683 5631 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9058f04895554d8585c606f129db9310" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:53:59.939631 5631 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:53:59.941489 5609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9058f04895554d8585c606f129db9310" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:53:59.941807 5609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: This master's current role is: LEADER
I20260620 01:53:59.969775 5637 catalog_manager.cc:1294] Loaded cluster ID: 406167b0b78a4a00b6eb7f76168e5158
I20260620 01:53:59.969924 5637 catalog_manager.cc:1587] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44: loading cluster ID for follower catalog manager: success
I20260620 01:53:59.974061 5637 catalog_manager.cc:1609] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44: acquiring CA information for follower catalog manager: success
I20260620 01:53:59.978184 5637 catalog_manager.cc:1637] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:54:00.029114 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.104s user 0.004s sys 0.017s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:00.112607 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:00.303709 5593 catalog_manager.cc:1294] Loaded cluster ID: 406167b0b78a4a00b6eb7f76168e5158
I20260620 01:54:00.304001 5593 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: loading cluster ID for follower catalog manager: success
I20260620 01:54:00.310767 5593 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: acquiring CA information for follower catalog manager: success
I20260620 01:54:00.315227 5593 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 01:54:00.639678 5642 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:00.640128 5642 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:00.640275 5642 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:00.651302 5642 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:00.651582 5642 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:54:00.670552 5642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:00.673094 5642 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:00.675460 5642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:00.690452 5647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:00.690650 5648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:00.691999 5650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:00.692507 5642 server_base.cc:1061] running on GCE node
I20260620 01:54:00.693336 5642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:00.694775 5642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:00.698392 5642 hybrid_clock.cc:648] HybridClock initialized: now 1781920440698259 us; error 98 us; skew 500 ppm
I20260620 01:54:00.699023 5642 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:00 dist-test-slave-wg14 krb5kdc[5332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920440, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:00.703965 5642 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:54:00.707101 5642 webserver.cc:533] Webserver started at http://127.20.1.129:35147/ using document root <none> and password file <none>
I20260620 01:54:00.708333 5642 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:00.708510 5642 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:00.709004 5642 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:54:00.712435 5642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "d0b25c1ac6d3440b986814453a81522c"
format_stamp: "Formatted at 2026-06-20 01:54:00 on dist-test-slave-wg14"
I20260620 01:54:00.713632 5642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "d0b25c1ac6d3440b986814453a81522c"
format_stamp: "Formatted at 2026-06-20 01:54:00 on dist-test-slave-wg14"
I20260620 01:54:00.721310 5642 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260620 01:54:00.727186 5657 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:00.729890 5642 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260620 01:54:00.730217 5642 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "d0b25c1ac6d3440b986814453a81522c"
format_stamp: "Formatted at 2026-06-20 01:54:00 on dist-test-slave-wg14"
I20260620 01:54:00.730477 5642 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:00.753305 5642 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:00.759895 5642 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:00.760249 5642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:00.761902 5642 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:00.764667 5642 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:00.764855 5642 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:00.765007 5642 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:00.765071 5642 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:00.811475 5642 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38997
I20260620 01:54:00.811559 5770 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38997 every 8 connection(s)
I20260620 01:54:00.813659 5642 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:54:00.823247 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 5642
I20260620 01:54:00.823830 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 01:54:00 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920440, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:54:00 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920440, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.130@KRBTEST.COM" created.
Jun 20 01:54:00 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920440, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:54:00.876005 5773 heartbeater.cc:344] Connected to a master server at 127.20.1.189:34883
I20260620 01:54:00.876060 5772 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44547
I20260620 01:54:00.876549 5773 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:00.876544 5772 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:00.878042 5772 heartbeater.cc:507] Master 127.20.1.190:44547 requested a full tablet report, sending...
I20260620 01:54:00.880054 5773 heartbeater.cc:507] Master 127.20.1.189:34883 requested a full tablet report, sending...
I20260620 01:54:00.881862 5380 ts_manager.cc:194] Registered new tserver with Master: d0b25c1ac6d3440b986814453a81522c (127.20.1.129:38997)
I20260620 01:54:00.882088 5457 ts_manager.cc:194] Registered new tserver with Master: d0b25c1ac6d3440b986814453a81522c (127.20.1.129:38997)
I20260620 01:54:00.884310 5771 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44059
I20260620 01:54:00.884537 5771 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:00.885365 5771 heartbeater.cc:507] Master 127.20.1.188:44059 requested a full tablet report, sending...
I20260620 01:54:00.888352 5537 ts_manager.cc:194] Registered new tserver with Master: d0b25c1ac6d3440b986814453a81522c (127.20.1.129:38997)
I20260620 01:54:00.890568 5537 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:59467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:00.972976 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:54:01.446214 5785 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:01.446687 5785 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:01.446861 5785 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:01.460170 5785 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:01.460500 5785 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:54:01.474596 5785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:01.476714 5785 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:01.478510 5785 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:01.492134 5790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:01.492018 5791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:01.492255 5793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:01.493974 5785 server_base.cc:1061] running on GCE node
I20260620 01:54:01.495114 5785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:01.496879 5785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:01.498242 5785 hybrid_clock.cc:648] HybridClock initialized: now 1781920441498149 us; error 77 us; skew 500 ppm
I20260620 01:54:01.498761 5785 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:01 dist-test-slave-wg14 krb5kdc[5332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920441, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:01.503484 5785 init.cc:377] Logged in from keytab as kudu/127.20.1.130@KRBTEST.COM (short username kudu)
I20260620 01:54:01.506733 5785 webserver.cc:533] Webserver started at http://127.20.1.130:40423/ using document root <none> and password file <none>
I20260620 01:54:01.508098 5785 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:01.508271 5785 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:01.508695 5785 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:54:01.512001 5785 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/data/instance:
uuid: "c669bcb3ef864811af0150ff4f216757"
format_stamp: "Formatted at 2026-06-20 01:54:01 on dist-test-slave-wg14"
I20260620 01:54:01.513484 5785 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance:
uuid: "c669bcb3ef864811af0150ff4f216757"
format_stamp: "Formatted at 2026-06-20 01:54:01 on dist-test-slave-wg14"
I20260620 01:54:01.522003 5785 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s
I20260620 01:54:01.529484 5800 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:01.532338 5785 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260620 01:54:01.532833 5785 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "c669bcb3ef864811af0150ff4f216757"
format_stamp: "Formatted at 2026-06-20 01:54:01 on dist-test-slave-wg14"
I20260620 01:54:01.533314 5785 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:01.575779 5785 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:01.581655 5785 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:01.582142 5785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:01.585242 5785 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:01.589188 5785 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:01.589334 5785 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:01.589473 5785 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:01.589543 5785 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:01.642747 5785 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:39445
I20260620 01:54:01.642984 5912 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:39445 every 8 connection(s)
I20260620 01:54:01.645056 5785 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:54:01.649923 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 5785
I20260620 01:54:01.650187 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
Jun 20 01:54:01 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920441, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:54:01 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920441, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:54:01 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920441, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.131@KRBTEST.COM; defaulting to no policy
I20260620 01:54:01.717871 5915 heartbeater.cc:344] Connected to a master server at 127.20.1.189:34883
I20260620 01:54:01.718146 5914 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44547
I20260620 01:54:01.718680 5915 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:01.718686 5914 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:01.719043 5913 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.131@KRBTEST.COM" created.
I20260620 01:54:01.719265 5913 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:01.720683 5915 heartbeater.cc:507] Master 127.20.1.189:34883 requested a full tablet report, sending...
I20260620 01:54:01.720789 5913 heartbeater.cc:507] Master 127.20.1.188:44059 requested a full tablet report, sending...
I20260620 01:54:01.720731 5914 heartbeater.cc:507] Master 127.20.1.190:44547 requested a full tablet report, sending...
I20260620 01:54:01.724331 5537 ts_manager.cc:194] Registered new tserver with Master: c669bcb3ef864811af0150ff4f216757 (127.20.1.130:39445)
I20260620 01:54:01.724874 5380 ts_manager.cc:194] Registered new tserver with Master: c669bcb3ef864811af0150ff4f216757 (127.20.1.130:39445)
I20260620 01:54:01.725965 5537 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:33721
I20260620 01:54:01.727715 5457 ts_manager.cc:194] Registered new tserver with Master: c669bcb3ef864811af0150ff4f216757 (127.20.1.130:39445)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:54:01.760208 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.131: real 0.100s user 0.000s sys 0.026s
WARNING: no policy specified for HTTP/127.20.1.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:01.833376 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:01.895279 5771 heartbeater.cc:499] Master 127.20.1.188:44059 was elected leader, sending a full tablet report...
W20260620 01:54:02.299708 5928 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:02.300239 5928 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:02.300413 5928 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:02.311762 5928 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:02.312039 5928 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:54:02.326145 5928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:02.328150 5928 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:02.330200 5928 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:02.347640 5934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:02.347920 5933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:02.348524 5936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:02.351075 5928 server_base.cc:1061] running on GCE node
I20260620 01:54:02.351969 5928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:02.353404 5928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:02.354856 5928 hybrid_clock.cc:648] HybridClock initialized: now 1781920442354710 us; error 106 us; skew 500 ppm
I20260620 01:54:02.355705 5928 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:02 dist-test-slave-wg14 krb5kdc[5332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920442, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:02.362627 5928 init.cc:377] Logged in from keytab as kudu/127.20.1.131@KRBTEST.COM (short username kudu)
I20260620 01:54:02.364970 5928 webserver.cc:533] Webserver started at http://127.20.1.131:40561/ using document root <none> and password file <none>
I20260620 01:54:02.366000 5928 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:02.366173 5928 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:02.366590 5928 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:54:02.369782 5928 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/data/instance:
uuid: "845c0605282142148150cc4c3dd75d66"
format_stamp: "Formatted at 2026-06-20 01:54:02 on dist-test-slave-wg14"
I20260620 01:54:02.372078 5928 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance:
uuid: "845c0605282142148150cc4c3dd75d66"
format_stamp: "Formatted at 2026-06-20 01:54:02 on dist-test-slave-wg14"
I20260620 01:54:02.381590 5928 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260620 01:54:02.387966 5943 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:02.390278 5928 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260620 01:54:02.390571 5928 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "845c0605282142148150cc4c3dd75d66"
format_stamp: "Formatted at 2026-06-20 01:54:02 on dist-test-slave-wg14"
I20260620 01:54:02.390826 5928 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:02.415481 5928 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:02.421921 5928 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:02.422349 5928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:02.423619 5928 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:02.425809 5928 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:02.425930 5928 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:02.426076 5928 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:02.426134 5928 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:02.480736 6057 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:37087 every 8 connection(s)
I20260620 01:54:02.480758 5928 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:37087
I20260620 01:54:02.483091 5928 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:54:02.490801 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 5928
I20260620 01:54:02.491097 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
Jun 20 01:54:02 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920442, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:54:02 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920442, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:54:02 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920442, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:54:02.528973 6059 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44547
I20260620 01:54:02.529151 6060 heartbeater.cc:344] Connected to a master server at 127.20.1.189:34883
I20260620 01:54:02.529217 6058 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44059
I20260620 01:54:02.529590 6058 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:02.529596 6060 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:02.529597 6059 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:02.531286 6060 heartbeater.cc:507] Master 127.20.1.189:34883 requested a full tablet report, sending...
I20260620 01:54:02.531314 6059 heartbeater.cc:507] Master 127.20.1.190:44547 requested a full tablet report, sending...
I20260620 01:54:02.531250 6058 heartbeater.cc:507] Master 127.20.1.188:44059 requested a full tablet report, sending...
I20260620 01:54:02.534015 5457 ts_manager.cc:194] Registered new tserver with Master: 845c0605282142148150cc4c3dd75d66 (127.20.1.131:37087)
I20260620 01:54:02.534238 5537 ts_manager.cc:194] Registered new tserver with Master: 845c0605282142148150cc4c3dd75d66 (127.20.1.131:37087)
I20260620 01:54:02.536026 5380 ts_manager.cc:194] Registered new tserver with Master: 845c0605282142148150cc4c3dd75d66 (127.20.1.131:37087)
I20260620 01:54:02.536029 5537 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:35875
I20260620 01:54:02.550124 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:54:02.564636 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:54:02.565078 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 5349
I20260620 01:54:02.566679 5411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:54:02.700333 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 5349
I20260620 01:54:02.726372 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44547
--webserver_interface=127.20.1.190
--webserver_port=33709
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:02.730955 5913 heartbeater.cc:499] Master 127.20.1.188:44059 was elected leader, sending a full tablet report...
W20260620 01:54:02.733664 5914 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:44547 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:44547: connect: Connection refused (error 111)
W20260620 01:54:02.889045 5772 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:44547 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:44547: connect: Connection refused (error 111)
W20260620 01:54:02.941277 5524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.190:44547: connect: Connection refused (error 111)
W20260620 01:54:02.944319 5524 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 -> Peer 4345ecdc1b104a6096e8744d2614ec04 (127.20.1.190:44547): Couldn't send request to peer 4345ecdc1b104a6096e8744d2614ec04. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:44547: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260620 01:54:03.204329 6070 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:03.204779 6070 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:03.204970 6070 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:03.218515 6070 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:54:03.218760 6070 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:03.218852 6070 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:54:03.218931 6070 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:54:03.233568 6070 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=33709
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:03.235589 6070 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:03.237370 6070 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:03.249454 6079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:03.249447 6078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:03.251565 6081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:03.252554 6070 server_base.cc:1061] running on GCE node
I20260620 01:54:03.253831 6070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:03.256096 6070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:03.257791 6070 hybrid_clock.cc:648] HybridClock initialized: now 1781920443257721 us; error 72 us; skew 500 ppm
I20260620 01:54:03.258340 6070 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:03 dist-test-slave-wg14 krb5kdc[5332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920443, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:03.263360 6070 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:54:03.265836 6070 webserver.cc:533] Webserver started at http://127.20.1.190:33709/ using document root <none> and password file <none>
I20260620 01:54:03.266849 6070 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:03.267037 6070 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:03.274755 6070 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:54:03.279799 6088 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:03.282924 6070 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260620 01:54:03.283308 6070 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "4345ecdc1b104a6096e8744d2614ec04"
format_stamp: "Formatted at 2026-06-20 01:53:57 on dist-test-slave-wg14"
I20260620 01:54:03.284852 6070 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:03.315550 6070 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:03.325706 6070 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:03.326061 6070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:03.356305 6070 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44547
I20260620 01:54:03.356325 6140 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44547 every 8 connection(s)
I20260620 01:54:03.358760 6070 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:54:03.366326 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 6070
I20260620 01:54:03.366847 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 5642
I20260620 01:54:03.370132 6141 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:54:03.375036 6141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: Bootstrap starting.
Jun 20 01:54:03 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 01:54:03.392221 5766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:54:03.398432 6141 log.cc:826] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: Log is configured to *not* fsync() on all Append() calls
I20260620 01:54:03.414320 6141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:54:03.415263 6141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: Bootstrap complete.
I20260620 01:54:03.428143 6141 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:54:03.429409 6141 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4345ecdc1b104a6096e8744d2614ec04, State: Initialized, Role: FOLLOWER
I20260620 01:54:03.430985 6141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:54:03.432473 6147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:03.432932 6147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:03.433817 6141 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:54:03.448096 6158 catalog_manager.cc:1294] Loaded cluster ID: 406167b0b78a4a00b6eb7f76168e5158
I20260620 01:54:03.448261 6158 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: loading cluster ID for follower catalog manager: success
I20260620 01:54:03.454695 6158 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: acquiring CA information for follower catalog manager: success
I20260620 01:54:03.460258 6158 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:54:03.523607 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 5642
I20260620 01:54:03.540174 6058 heartbeater.cc:499] Master 127.20.1.188:44059 was elected leader, sending a full tablet report...
I20260620 01:54:03.556685 6163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9058f04895554d8585c606f129db9310" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:03.558923 6163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: This master's current role is: LEADER
I20260620 01:54:03.566617 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:38997
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--webserver_port=35147
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:03.569687 6106 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "845c0605282142148150cc4c3dd75d66" instance_seqno: 1781920442465897) as {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:47357; Asking this server to re-register.
I20260620 01:54:03.571683 6059 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:03.574709 6106 ts_manager.cc:194] Registered new tserver with Master: 845c0605282142148150cc4c3dd75d66 (127.20.1.131:37087)
I20260620 01:54:03.575752 6147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9058f04895554d8585c606f129db9310. Latest consensus state: current_term: 1 leader_uuid: "9058f04895554d8585c606f129db9310" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:03.576113 6147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:03.748338 5914 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44547
I20260620 01:54:03.750590 6106 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c669bcb3ef864811af0150ff4f216757" instance_seqno: 1781920441630245) as {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:45273; Asking this server to re-register.
I20260620 01:54:03.752055 5914 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:03.754494 6106 ts_manager.cc:194] Registered new tserver with Master: c669bcb3ef864811af0150ff4f216757 (127.20.1.130:39445)
W20260620 01:54:04.054888 6169 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:04.055471 6169 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:04.055895 6169 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:04.067646 6169 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:04.067956 6169 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:54:04.076583 6163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9058f04895554d8585c606f129db9310" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:04.076977 6163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: This master's current role is: LEADER
I20260620 01:54:04.083783 6169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:38997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=35147
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:04.085847 6169 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:04.087790 6169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:04.103652 6179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:04.103542 6178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:04.103608 6181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:04.104538 6169 server_base.cc:1061] running on GCE node
I20260620 01:54:04.105707 6169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:04.107332 6169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:04.108902 6169 hybrid_clock.cc:648] HybridClock initialized: now 1781920444108663 us; error 91 us; skew 500 ppm
I20260620 01:54:04.109397 6169 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:04 dist-test-slave-wg14 krb5kdc[5332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920444, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:04.116341 6169 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:54:04.119570 6169 webserver.cc:533] Webserver started at http://127.20.1.129:35147/ using document root <none> and password file <none>
I20260620 01:54:04.120676 6169 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:04.120886 6169 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:04.129773 6169 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 01:54:04.136291 6188 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:04.139746 6169 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.006s sys 0.000s
I20260620 01:54:04.140034 6169 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "d0b25c1ac6d3440b986814453a81522c"
format_stamp: "Formatted at 2026-06-20 01:54:00 on dist-test-slave-wg14"
I20260620 01:54:04.141462 6169 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:04.164767 6169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:04.171108 6169 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:04.171722 6169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:04.173552 6169 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:04.176661 6169 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:04.176882 6169 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:04.177018 6169 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:04.177114 6169 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:04.228816 6169 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38997
I20260620 01:54:04.228943 6301 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38997 every 8 connection(s)
I20260620 01:54:04.232128 6169 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:54:04.236704 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 6169
Jun 20 01:54:04 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920444, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:54:04 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920444, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:54:04 dist-test-slave-wg14 krb5kdc[5332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920444, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:54:04.282791 6304 heartbeater.cc:344] Connected to a master server at 127.20.1.189:34883
I20260620 01:54:04.283555 6304 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:04.285326 6304 heartbeater.cc:507] Master 127.20.1.189:34883 requested a full tablet report, sending...
I20260620 01:54:04.286577 6303 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44547
I20260620 01:54:04.286808 6303 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:04.287518 6303 heartbeater.cc:507] Master 127.20.1.190:44547 requested a full tablet report, sending...
I20260620 01:54:04.290021 6106 ts_manager.cc:194] Registered new tserver with Master: d0b25c1ac6d3440b986814453a81522c (127.20.1.129:38997)
I20260620 01:54:04.291241 5457 ts_manager.cc:194] Re-registered known tserver with Master: d0b25c1ac6d3440b986814453a81522c (127.20.1.129:38997)
I20260620 01:54:04.292044 6302 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44059
I20260620 01:54:04.292286 6302 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:04.293094 6302 heartbeater.cc:507] Master 127.20.1.188:44059 requested a full tablet report, sending...
I20260620 01:54:04.296128 5537 ts_manager.cc:194] Re-registered known tserver with Master: d0b25c1ac6d3440b986814453a81522c (127.20.1.129:38997)
I20260620 01:54:04.297410 5537 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:35929
I20260620 01:54:04.315207 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 6169
W20260620 01:54:04.316996 6314 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:04.311226 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.20.1.129:38997)
0620 01:54:04.312439 (+ 1213us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:04.312452 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:04.312499 (+ 47us) server_negotiation.cc:408] Connection header received
0620 01:54:04.312593 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:04.312602 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:04.313044 (+ 442us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:04.313322 (+ 278us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:04.314728 (+ 1406us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60298: BlockingRecv error: recv got EOF from 127.0.0.1:60298 (error 108)
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":553,"threads_started":1}
W20260620 01:54:04.319288 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 6169: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:38997: server requires authentication, but client does not have Kerberos credentials available
W20260620 01:54:04.319398 6315 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:04.316672 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:60310 (local address 127.20.1.129:38997)
0620 01:54:04.317616 (+ 944us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:04.317622 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:04.317648 (+ 26us) server_negotiation.cc:408] Connection header received
0620 01:54:04.317730 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:04.317735 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:04.317818 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:04.317941 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:04.319027 (+ 1086us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60310: BlockingRecv error: recv got EOF from 127.0.0.1:60310 (error 108)
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":526,"threads_started":1}
I20260620 01:54:04.319567 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 6169
I20260620 01:54:04.335774 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:38997
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--webserver_port=35147
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:54:04.784835 6316 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:04.785286 6316 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:04.785398 6316 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:54:04.785462 6316 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:04.796032 6316 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:04.796288 6316 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:54:04.809405 6316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:38997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=35147
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:04.812019 6316 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:04.814292 6316 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:04.829625 6321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:04.829730 6324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:04.829547 6322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:04.832075 6316 server_base.cc:1061] running on GCE node
I20260620 01:54:04.833491 6316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:04.835368 6316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:04.836814 6316 hybrid_clock.cc:648] HybridClock initialized: now 1781920444836674 us; error 82 us; skew 500 ppm
I20260620 01:54:04.837394 6316 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:04 dist-test-slave-wg14 krb5kdc[5332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920444, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:04.842862 6316 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:54:04.845734 6316 webserver.cc:533] Webserver started at http://127.20.1.129:35147/ using document root <none> and password file <none>
I20260620 01:54:04.846854 6316 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:04.847021 6316 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:04.855971 6316 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 01:54:04.860533 6331 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:04.862665 6316 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:54:04.862898 6316 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "d0b25c1ac6d3440b986814453a81522c"
format_stamp: "Formatted at 2026-06-20 01:54:00 on dist-test-slave-wg14"
I20260620 01:54:04.863996 6316 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:04.892060 6316 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:04.898319 6316 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:04.898681 6316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:04.899982 6316 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:04.901997 6316 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:04.902110 6316 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:04.902216 6316 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:04.902278 6316 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:54:04.903024 6316 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:54:04.903205 6316 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:54:04.903297 6316 fault_injection.cc:59] injecting fault for kudu (pid 6316): FLAGS_fault_before_start (process will exit)
I20260620 01:54:04.920866 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:38997
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--webserver_port=35147
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:54:05.377149 6339 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:05.377585 6339 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:05.377748 6339 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:05.388285 6339 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:05.388602 6339 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:54:05.401440 6339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:38997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=35147
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:05.403529 6339 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:05.405341 6339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:05.419004 6344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:05.419049 6345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:05.421442 6347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:05.422211 6339 server_base.cc:1061] running on GCE node
I20260620 01:54:05.423026 6339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:05.424357 6339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:05.428016 6339 hybrid_clock.cc:648] HybridClock initialized: now 1781920445427571 us; error 293 us; skew 500 ppm
I20260620 01:54:05.428781 6339 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:05 dist-test-slave-wg14 krb5kdc[5332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920445, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:05.433789 6339 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:54:05.437127 6339 webserver.cc:533] Webserver started at http://127.20.1.129:35147/ using document root <none> and password file <none>
I20260620 01:54:05.438575 6339 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:05.438786 6339 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:05.447827 6339 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 01:54:05.452343 6354 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:05.454563 6339 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:54:05.454843 6339 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "d0b25c1ac6d3440b986814453a81522c"
format_stamp: "Formatted at 2026-06-20 01:54:00 on dist-test-slave-wg14"
I20260620 01:54:05.456223 6339 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:05.480516 6339 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:05.486047 6339 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:05.486413 6339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:05.488001 6339 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:05.490556 6339 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:05.491079 6339 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260620 01:54:05.491269 6339 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:05.491348 6339 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:05.543260 6339 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38997
I20260620 01:54:05.543445 6467 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38997 every 8 connection(s)
I20260620 01:54:05.545729 6339 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:54:05.552186 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 6339
I20260620 01:54:05.552672 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 6339
W20260620 01:54:05.569410 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 6339: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:38997: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:54:05.569638 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 6339
I20260620 01:54:05.587836 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 5785
W20260620 01:54:05.588618 6473 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:05.551213 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:56841 (local address 127.20.1.190:44547)
0620 01:54:05.560742 (+ 9529us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:05.560748 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:05.587561 (+ 26813us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:56841: BlockingRecv error: recv got EOF from 127.20.1.129:56841 (error 108)
Metrics: {"server-negotiator.queue_time_us":9360,"thread_start_us":9235,"threads_started":1}
W20260620 01:54:05.589290 6472 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:05.550788 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:44501 (local address 127.20.1.188:44059)
0620 01:54:05.563946 (+ 13158us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:05.563955 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:05.587772 (+ 23817us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:44501: BlockingRecv error: recv got EOF from 127.20.1.129:44501 (error 108)
Metrics: {"server-negotiator.queue_time_us":12975,"thread_start_us":12675,"threads_started":1}
W20260620 01:54:05.592060 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 5785: Not authorized: Client connection negotiation failed: client connection to 127.20.1.130:39445: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:54:05.592342 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 5785
I20260620 01:54:05.608806 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 5928
W20260620 01:54:05.613305 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 5928: Not authorized: Client connection negotiation failed: client connection to 127.20.1.131:37087: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:54:05.613675 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 5928
I20260620 01:54:05.630007 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 6070
W20260620 01:54:05.632866 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 6070: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:44547: server requires authentication, but client does not have Kerberos credentials available
W20260620 01:54:05.633010 6473 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:05.631281 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.20.1.190:44547)
0620 01:54:05.631523 (+ 242us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:05.631528 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:05.631549 (+ 21us) server_negotiation.cc:408] Connection header received
0620 01:54:05.631619 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:05.631624 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:05.631704 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:05.631837 (+ 133us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:05.632762 (+ 925us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34252: BlockingRecv error: recv got EOF from 127.0.0.1:34252 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 01:54:05.633049 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 6070
I20260620 01:54:05.655068 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 5425
I20260620 01:54:05.657053 5488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:54:05.787657 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 5425
I20260620 01:54:05.808684 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 5503
I20260620 01:54:05.810562 5567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:54:05.941311 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 5503
2026-06-20T01:54:05Z chronyd exiting
Jun 20 01:54:05 dist-test-slave-wg14 krb5kdc[6487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 01:54:06 dist-test-slave-wg14 krb5kdc[6487](info): set up 2 sockets
Jun 20 01:54:06 dist-test-slave-wg14 krb5kdc[6487](info): commencing operation
krb5kdc: starting...
W20260620 01:54:07.952872 20486 mini_kdc.cc:121] Time spent starting KDC: real 1.974s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 01:54:08 dist-test-slave-wg14 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T01:54:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:54:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.190@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.190@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:08.271966 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44547
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:54:08.780241 6503 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:08.780932 6503 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:08.781150 6503 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:08.792095 6503 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:54:08.792259 6503 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:08.792330 6503 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:54:08.792378 6503 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:54:08.806613 6503 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:08.809065 6503 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:08.810954 6503 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:08.823320 6508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:08.823590 6509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:08.824820 6511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:08.827499 6503 server_base.cc:1061] running on GCE node
I20260620 01:54:08.828763 6503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:08.831362 6503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:08.832830 6503 hybrid_clock.cc:648] HybridClock initialized: now 1781920448832690 us; error 137 us; skew 500 ppm
I20260620 01:54:08.833458 6503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:08 dist-test-slave-wg14 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920448, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:08.838860 6503 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:54:08.842386 6503 webserver.cc:533] Webserver started at http://127.20.1.190:46877/ using document root <none> and password file <none>
I20260620 01:54:08.843772 6503 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:08.843951 6503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:08.853323 6503 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 01:54:08.858366 6518 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:08.861325 6503 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:54:08.861624 6503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "4345ecdc1b104a6096e8744d2614ec04"
format_stamp: "Formatted at 2026-06-20 01:53:57 on dist-test-slave-wg14"
I20260620 01:54:08.862686 6503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:08.882604 6503 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:08.894424 6503 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:08.894757 6503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:08.926401 6503 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44547
I20260620 01:54:08.926680 6570 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44547 every 8 connection(s)
I20260620 01:54:08.928751 6503 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:54:08.930832 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 6503
I20260620 01:54:08.931177 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:54:08.939067 6571 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:54:08.946173 6571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: Bootstrap starting.
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.189@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:54:08.977540 6571 log.cc:826] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: Log is configured to *not* fsync() on all Append() calls
I20260620 01:54:08.994886 6571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:54:08.995879 6571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: Bootstrap complete.
I20260620 01:54:09.010231 6571 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:54:09.011566 6571 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4345ecdc1b104a6096e8744d2614ec04, State: Initialized, Role: FOLLOWER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:09.012779 6571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:54:09.014101 6576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:09.014593 6576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:09.015620 6571 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:54:09.045836 6588 catalog_manager.cc:1294] Loaded cluster ID: 406167b0b78a4a00b6eb7f76168e5158
I20260620 01:54:09.046132 6588 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: loading cluster ID for follower catalog manager: success
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.189@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:54:09.053299 6588 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: acquiring CA information for follower catalog manager: success
I20260620 01:54:09.057454 6588 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Jun 20 01:54:09 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920448, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:54:09.083356 6503 master_runner.cc:187] Error getting master registration for 127.20.1.189:34883: Network error: Client connection negotiation failed: client connection to 127.20.1.189:34883: connect: Connection refused (error 111),
I20260620 01:54:09.090348 6503 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:09.099345 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:34883
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:54:09.565927 6593 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:09.566350 6593 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:09.566501 6593 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:09.578825 6593 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:54:09.578991 6593 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:09.579054 6593 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:54:09.579170 6593 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:54:09.593003 6593 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:34883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:09.596140 6593 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:09.598358 6593 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:09.612020 6601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:09.612013 6599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:09.612056 6598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:09.613514 6593 server_base.cc:1061] running on GCE node
I20260620 01:54:09.614607 6593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:09.616921 6593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:09.618431 6593 hybrid_clock.cc:648] HybridClock initialized: now 1781920449618338 us; error 75 us; skew 500 ppm
I20260620 01:54:09.619094 6593 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:09 dist-test-slave-wg14 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920449, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:09.624692 6593 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 01:54:09.627509 6593 webserver.cc:533] Webserver started at http://127.20.1.189:35389/ using document root <none> and password file <none>
I20260620 01:54:09.628686 6593 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:09.628906 6593 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:09.637697 6593 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 01:54:09.642135 6608 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:09.644510 6593 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260620 01:54:09.644788 6593 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "cd052da5773244dd80b5fbbd78139b44"
format_stamp: "Formatted at 2026-06-20 01:53:58 on dist-test-slave-wg14"
I20260620 01:54:09.645855 6593 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:09.672693 6593 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:09.678404 6593 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:09.678704 6593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:09.706959 6593 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:34883
I20260620 01:54:09.706983 6659 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:34883 every 8 connection(s)
I20260620 01:54:09.709429 6593 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:54:09.714315 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 6593
I20260620 01:54:09.714564 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:54:09.719542 6660 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:54:09.724339 6660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44: Bootstrap starting.
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.188@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:54:09.750372 6660 log.cc:826] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44: Log is configured to *not* fsync() on all Append() calls
I20260620 01:54:09.766302 6660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:54:09.767299 6660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44: Bootstrap complete.
I20260620 01:54:09.779789 6660 raft_consensus.cc:359] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:54:09.780987 6660 raft_consensus.cc:740] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd052da5773244dd80b5fbbd78139b44, State: Initialized, Role: FOLLOWER
I20260620 01:54:09.782136 6660 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:54:09.783493 6665 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:09.783943 6665 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:09.785022 6660 sys_catalog.cc:565] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: configured and running, proceeding with master startup.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:09.815783 6676 catalog_manager.cc:1294] Loaded cluster ID: 406167b0b78a4a00b6eb7f76168e5158
I20260620 01:54:09.816172 6676 catalog_manager.cc:1587] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44: loading cluster ID for follower catalog manager: success
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.188@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:54:09.823932 6676 catalog_manager.cc:1609] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44: acquiring CA information for follower catalog manager: success
I20260620 01:54:09.828199 6676 catalog_manager.cc:1637] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Jun 20 01:54:09 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920449, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:09.869997 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:44059
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:09.883163 6593 master_runner.cc:187] Error getting master registration for 127.20.1.188:44059: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44059: connect: Connection refused (error 111),
I20260620 01:54:09.890806 6593 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:54:10.396257 6683 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:10.396752 6683 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:10.396900 6683 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:10.406688 6683 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:54:10.406858 6683 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:10.406939 6683 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:54:10.407001 6683 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:54:10.421489 6683 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:10.423573 6683 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:10.425411 6683 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:10.437247 6689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:10.437852 6690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:10.439505 6692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:10.439751 6683 server_base.cc:1061] running on GCE node
I20260620 01:54:10.440954 6683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:10.443220 6683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:10.445055 6683 hybrid_clock.cc:648] HybridClock initialized: now 1781920450444933 us; error 86 us; skew 500 ppm
I20260620 01:54:10.445720 6683 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:10 dist-test-slave-wg14 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920450, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:10.450356 6683 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 01:54:10.452891 6683 webserver.cc:533] Webserver started at http://127.20.1.188:34009/ using document root <none> and password file <none>
I20260620 01:54:10.453902 6683 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:10.454090 6683 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:10.461366 6683 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.005s
I20260620 01:54:10.465574 6699 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:10.468345 6683 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:54:10.468667 6683 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "9058f04895554d8585c606f129db9310"
format_stamp: "Formatted at 2026-06-20 01:53:59 on dist-test-slave-wg14"
I20260620 01:54:10.469702 6683 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:10.493983 6683 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:10.501010 6683 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:10.501368 6683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:10.530819 6683 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:44059
I20260620 01:54:10.530846 6750 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:44059 every 8 connection(s)
I20260620 01:54:10.532950 6683 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:54:10.540745 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 6683
I20260620 01:54:10.540968 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:54:10.543727 6751 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:54:10.550843 6751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310: Bootstrap starting.
Jun 20 01:54:10 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:54:10 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:54:10.576844 6751 log.cc:826] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310: Log is configured to *not* fsync() on all Append() calls
I20260620 01:54:10.591893 6751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:54:10.592711 6751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310: Bootstrap complete.
Jun 20 01:54:10 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:54:10.603245 6751 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:54:10.604332 6751 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9058f04895554d8585c606f129db9310, State: Initialized, Role: FOLLOWER
I20260620 01:54:10.605357 6751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:54:10.606626 6758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:10.607184 6758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:10.608270 6751 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:54:10.627638 6770 catalog_manager.cc:1294] Loaded cluster ID: 406167b0b78a4a00b6eb7f76168e5158
I20260620 01:54:10.627867 6770 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310: loading cluster ID for follower catalog manager: success
I20260620 01:54:10.633494 6770 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310: acquiring CA information for follower catalog manager: success
Jun 20 01:54:10 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920450, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:54:10.637586 6770 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:54:10.645948 6773 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:54:10.646343 6773 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:54:10.657431 6773 leader_election.cc:290] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd052da5773244dd80b5fbbd78139b44 (127.20.1.189:34883), 9058f04895554d8585c606f129db9310 (127.20.1.188:44059)
I20260620 01:54:10.677889 6635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4345ecdc1b104a6096e8744d2614ec04" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd052da5773244dd80b5fbbd78139b44" is_pre_election: true
I20260620 01:54:10.678889 6635 raft_consensus.cc:2468] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4345ecdc1b104a6096e8744d2614ec04 in term 1.
I20260620 01:54:10.680297 6521 leader_election.cc:304] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4345ecdc1b104a6096e8744d2614ec04, cd052da5773244dd80b5fbbd78139b44; no voters:
I20260620 01:54:10.681107 6773 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:54:10.681345 6773 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:54:10.681473 6773 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:54:10.687956 6773 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:54:10.689182 6773 leader_election.cc:290] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [CANDIDATE]: Term 2 election: Requested vote from peers cd052da5773244dd80b5fbbd78139b44 (127.20.1.189:34883), 9058f04895554d8585c606f129db9310 (127.20.1.188:44059)
I20260620 01:54:10.690418 6635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4345ecdc1b104a6096e8744d2614ec04" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd052da5773244dd80b5fbbd78139b44"
I20260620 01:54:10.690724 6635 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:54:10.693181 6726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4345ecdc1b104a6096e8744d2614ec04" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9058f04895554d8585c606f129db9310" is_pre_election: true
I20260620 01:54:10.694039 6726 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4345ecdc1b104a6096e8744d2614ec04 in term 1.
I20260620 01:54:10.696170 6726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4345ecdc1b104a6096e8744d2614ec04" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9058f04895554d8585c606f129db9310"
I20260620 01:54:10.696520 6726 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:54:10.697484 6635 raft_consensus.cc:2468] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4345ecdc1b104a6096e8744d2614ec04 in term 2.
I20260620 01:54:10.698474 6521 leader_election.cc:304] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4345ecdc1b104a6096e8744d2614ec04, cd052da5773244dd80b5fbbd78139b44; no voters:
I20260620 01:54:10.700724 6773 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 01:54:10.704604 6726 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4345ecdc1b104a6096e8744d2614ec04 in term 2.
I20260620 01:54:10.708604 6683 master_runner.cc:264] No leader master found from master 9058f04895554d8585c606f129db9310
I20260620 01:54:10.708832 6683 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:54:10.711849 6773 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 2 LEADER]: Becoming Leader. State: Replica: 4345ecdc1b104a6096e8744d2614ec04, State: Running, Role: LEADER
I20260620 01:54:10.712735 6773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:54:10.717787 6777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4345ecdc1b104a6096e8744d2614ec04. Latest consensus state: current_term: 2 leader_uuid: "4345ecdc1b104a6096e8744d2614ec04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:10.718273 6777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: This master's current role is: LEADER
I20260620 01:54:10.720907 6779 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:54:10.724265 6779 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:54:10.725688 6779 catalog_manager.cc:1294] Loaded cluster ID: 406167b0b78a4a00b6eb7f76168e5158
I20260620 01:54:10.725863 6779 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:54:10.727751 6779 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:54:10.729202 6779 catalog_manager.cc:6080] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: Loaded TSK: 0
I20260620 01:54:10.730859 6779 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.129@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.129@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:10.855693 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:11.274009 6635 raft_consensus.cc:1275] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 2 FOLLOWER]: Refusing update from remote peer 4345ecdc1b104a6096e8744d2614ec04: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:54:11.276319 6785 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:54:11.282116 6786 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4345ecdc1b104a6096e8744d2614ec04. Latest consensus state: current_term: 2 leader_uuid: "4345ecdc1b104a6096e8744d2614ec04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:11.282565 6786 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:11.285035 6785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4345ecdc1b104a6096e8744d2614ec04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:11.285422 6785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: This master's current role is: LEADER
I20260620 01:54:11.288131 6726 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 2 FOLLOWER]: Refusing update from remote peer 4345ecdc1b104a6096e8744d2614ec04: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:54:11.291483 6788 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:54:11.300931 6786 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4345ecdc1b104a6096e8744d2614ec04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:11.301945 6786 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:11.305215 6793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4345ecdc1b104a6096e8744d2614ec04. Latest consensus state: current_term: 2 leader_uuid: "4345ecdc1b104a6096e8744d2614ec04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:11.305514 6793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:11.305581 6796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4345ecdc1b104a6096e8744d2614ec04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:11.305987 6796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:54:11.335062 6784 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:11.335561 6784 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:11.335695 6784 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:11.347294 6784 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:11.347522 6784 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:54:11.362299 6784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:11.364594 6784 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:11.366354 6784 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:11.380056 6802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:11.380144 6801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:11.381134 6804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:11.383118 6784 server_base.cc:1061] running on GCE node
I20260620 01:54:11.384398 6784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:11.386610 6784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:11.388069 6784 hybrid_clock.cc:648] HybridClock initialized: now 1781920451387919 us; error 127 us; skew 500 ppm
I20260620 01:54:11.388592 6784 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:11 dist-test-slave-wg14 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920451, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:11.393139 6784 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:54:11.396016 6784 webserver.cc:533] Webserver started at http://127.20.1.129:44777/ using document root <none> and password file <none>
I20260620 01:54:11.397182 6784 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:11.397382 6784 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:11.407217 6784 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:54:11.411939 6811 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:11.414922 6784 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.000s sys 0.004s
I20260620 01:54:11.415266 6784 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "d0b25c1ac6d3440b986814453a81522c"
format_stamp: "Formatted at 2026-06-20 01:54:00 on dist-test-slave-wg14"
I20260620 01:54:11.416266 6784 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:11.441378 6784 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:11.446507 6784 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:11.446936 6784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:11.448871 6784 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:11.451406 6784 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:11.451571 6784 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:11.451692 6784 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:11.451767 6784 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:11.497289 6784 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35207
I20260620 01:54:11.497390 6924 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35207 every 8 connection(s)
I20260620 01:54:11.499522 6784 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:54:11.509897 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 6784
I20260620 01:54:11.510198 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 01:54:11 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920451, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:54:11 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920451, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:54:11 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920451, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:54:11.542665 6925 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44059
I20260620 01:54:11.543793 6925 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:11.545163 6925 heartbeater.cc:507] Master 127.20.1.188:44059 requested a full tablet report, sending...
I20260620 01:54:11.548982 6715 ts_manager.cc:194] Registered new tserver with Master: d0b25c1ac6d3440b986814453a81522c (127.20.1.129:35207)
I20260620 01:54:11.553444 6926 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44547
I20260620 01:54:11.553707 6926 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:11.554330 6926 heartbeater.cc:507] Master 127.20.1.190:44547 requested a full tablet report, sending...
I20260620 01:54:11.556129 6927 heartbeater.cc:344] Connected to a master server at 127.20.1.189:34883
I20260620 01:54:11.556388 6927 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:11.557008 6927 heartbeater.cc:507] Master 127.20.1.189:34883 requested a full tablet report, sending...
I20260620 01:54:11.557269 6535 ts_manager.cc:194] Registered new tserver with Master: d0b25c1ac6d3440b986814453a81522c (127.20.1.129:35207)
I20260620 01:54:11.559793 6535 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:34449
I20260620 01:54:11.562326 6625 ts_manager.cc:194] Registered new tserver with Master: d0b25c1ac6d3440b986814453a81522c (127.20.1.129:35207)
WARNING: no policy specified for kudu/127.20.1.130@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.130@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.130 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.130 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.130@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.130@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.130 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.130 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:11.656368 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:11.868467 6940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4345ecdc1b104a6096e8744d2614ec04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:11.868921 6940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: This master's current role is: LEADER
W20260620 01:54:12.109004 6939 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:12.109743 6939 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:12.109968 6939 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:12.120679 6939 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:12.120908 6939 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:54:12.133813 6939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:12.135862 6939 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:12.137573 6939 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:12.151026 6948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:12.151238 6949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:12.151283 6951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:12.152599 6939 server_base.cc:1061] running on GCE node
I20260620 01:54:12.153530 6939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:12.154997 6939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:12.156497 6939 hybrid_clock.cc:648] HybridClock initialized: now 1781920452156424 us; error 69 us; skew 500 ppm
I20260620 01:54:12.157290 6939 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:12 dist-test-slave-wg14 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920452, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:12.162117 6939 init.cc:377] Logged in from keytab as kudu/127.20.1.130@KRBTEST.COM (short username kudu)
I20260620 01:54:12.165084 6939 webserver.cc:533] Webserver started at http://127.20.1.130:40979/ using document root <none> and password file <none>
I20260620 01:54:12.166115 6939 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:12.166325 6939 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:12.174986 6939 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:54:12.180052 6958 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:12.182605 6939 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:54:12.182897 6939 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "c669bcb3ef864811af0150ff4f216757"
format_stamp: "Formatted at 2026-06-20 01:54:01 on dist-test-slave-wg14"
I20260620 01:54:12.183905 6939 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:12.206545 6939 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:12.212517 6939 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:12.212910 6939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:12.214244 6939 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:12.216693 6939 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:12.217018 6939 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:12.217197 6939 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:12.217288 6939 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:12.264954 6939 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:40053
I20260620 01:54:12.265074 7071 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:40053 every 8 connection(s)
I20260620 01:54:12.266987 6939 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:54:12.269424 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 6939
I20260620 01:54:12.269767 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
Jun 20 01:54:12 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920452, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:54:12 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920452, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.131@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.131@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Jun 20 01:54:12 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920452, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:54:12.327587 7073 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44547
I20260620 01:54:12.327615 7072 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44059
I20260620 01:54:12.328217 7072 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:12.328225 7073 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:12.330055 7072 heartbeater.cc:507] Master 127.20.1.188:44059 requested a full tablet report, sending...
I20260620 01:54:12.330098 7073 heartbeater.cc:507] Master 127.20.1.190:44547 requested a full tablet report, sending...
I20260620 01:54:12.333197 6535 ts_manager.cc:194] Registered new tserver with Master: c669bcb3ef864811af0150ff4f216757 (127.20.1.130:40053)
I20260620 01:54:12.333362 6715 ts_manager.cc:194] Registered new tserver with Master: c669bcb3ef864811af0150ff4f216757 (127.20.1.130:40053)
I20260620 01:54:12.334985 6535 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:47889
I20260620 01:54:12.339177 7074 heartbeater.cc:344] Connected to a master server at 127.20.1.189:34883
I20260620 01:54:12.339428 7074 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:12.339931 7074 heartbeater.cc:507] Master 127.20.1.189:34883 requested a full tablet report, sending...
I20260620 01:54:12.341962 6625 ts_manager.cc:194] Registered new tserver with Master: c669bcb3ef864811af0150ff4f216757 (127.20.1.130:40053)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.131 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.131 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.131@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.131@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.131 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.131 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:12.428303 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:12.564131 6926 heartbeater.cc:499] Master 127.20.1.190:44547 was elected leader, sending a full tablet report...
W20260620 01:54:12.868988 7087 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:12.869468 7087 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:12.869625 7087 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:12.880136 7087 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:12.880471 7087 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:54:12.894335 7087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:12.896502 7087 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:12.898281 7087 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:12.914377 7093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:12.914414 7092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:12.916115 7095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:12.916678 7087 server_base.cc:1061] running on GCE node
I20260620 01:54:12.917701 7087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:12.919386 7087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:12.921051 7087 hybrid_clock.cc:648] HybridClock initialized: now 1781920452920962 us; error 83 us; skew 500 ppm
I20260620 01:54:12.921701 7087 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:12 dist-test-slave-wg14 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920452, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:12.927511 7087 init.cc:377] Logged in from keytab as kudu/127.20.1.131@KRBTEST.COM (short username kudu)
I20260620 01:54:12.930536 7087 webserver.cc:533] Webserver started at http://127.20.1.131:42899/ using document root <none> and password file <none>
I20260620 01:54:12.931850 7087 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:12.932072 7087 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:12.942242 7087 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.004s sys 0.004s
I20260620 01:54:12.947626 7102 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:12.950238 7087 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260620 01:54:12.950512 7087 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "845c0605282142148150cc4c3dd75d66"
format_stamp: "Formatted at 2026-06-20 01:54:02 on dist-test-slave-wg14"
I20260620 01:54:12.951493 7087 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:12.977114 7087 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:12.983610 7087 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:12.983982 7087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:12.985692 7087 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:12.988423 7087 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:12.988637 7087 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:12.988786 7087 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:12.988881 7087 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:13.037461 7087 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:45221
I20260620 01:54:13.037523 7215 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:45221 every 8 connection(s)
I20260620 01:54:13.040288 7087 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:54:13.044091 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 7087
I20260620 01:54:13.044903 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
Jun 20 01:54:13 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920452, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:54:13 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920452, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:54:13 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920452, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:54:13.087966 7218 heartbeater.cc:344] Connected to a master server at 127.20.1.189:34883
I20260620 01:54:13.088531 7218 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:13.090051 7218 heartbeater.cc:507] Master 127.20.1.189:34883 requested a full tablet report, sending...
I20260620 01:54:13.091169 7217 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44547
I20260620 01:54:13.091416 7217 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:13.091461 7216 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44059
I20260620 01:54:13.091763 7216 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:13.092314 7217 heartbeater.cc:507] Master 127.20.1.190:44547 requested a full tablet report, sending...
I20260620 01:54:13.092603 7216 heartbeater.cc:507] Master 127.20.1.188:44059 requested a full tablet report, sending...
I20260620 01:54:13.093185 6625 ts_manager.cc:194] Registered new tserver with Master: 845c0605282142148150cc4c3dd75d66 (127.20.1.131:45221)
I20260620 01:54:13.094272 6715 ts_manager.cc:194] Registered new tserver with Master: 845c0605282142148150cc4c3dd75d66 (127.20.1.131:45221)
I20260620 01:54:13.094296 6535 ts_manager.cc:194] Registered new tserver with Master: 845c0605282142148150cc4c3dd75d66 (127.20.1.131:45221)
I20260620 01:54:13.095537 6535 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:58861
I20260620 01:54:13.112370 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:54:13.128199 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:54:13.128487 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 6503
I20260620 01:54:13.130239 6566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:54:13.266870 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 6503
W20260620 01:54:13.297426 6700 connection.cc:573] server connection from 127.0.0.1:53782 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:54:13.299646 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44547
--webserver_interface=127.20.1.190
--webserver_port=46877
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:13.339536 7073 heartbeater.cc:499] Master 127.20.1.190:44547 was elected leader, sending a full tablet report...
W20260620 01:54:13.343174 7073 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:44547 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:44547: connect: Connection refused (error 111)
W20260620 01:54:13.569195 6926 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:44547 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:44547: connect: Connection refused (error 111)
W20260620 01:54:13.782914 7227 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:13.783408 7227 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:13.783522 7227 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:13.794155 7227 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:54:13.794298 7227 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:13.794360 7227 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:54:13.794401 7227 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:54:13.807303 7227 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=46877
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:13.809376 7227 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:13.811929 7227 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:13.824518 7237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:13.824537 7234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:13.824628 7235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:13.826596 7227 server_base.cc:1061] running on GCE node
I20260620 01:54:13.828119 7227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:13.830246 7227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:13.831625 7227 hybrid_clock.cc:648] HybridClock initialized: now 1781920453831541 us; error 75 us; skew 500 ppm
I20260620 01:54:13.832150 7227 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:13 dist-test-slave-wg14 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920453, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:13.837121 7227 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:54:13.839766 7227 webserver.cc:533] Webserver started at http://127.20.1.190:46877/ using document root <none> and password file <none>
I20260620 01:54:13.841177 7227 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:13.841361 7227 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:13.849193 7227 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 01:54:13.854143 7244 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:13.856607 7227 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:54:13.856935 7227 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "4345ecdc1b104a6096e8744d2614ec04"
format_stamp: "Formatted at 2026-06-20 01:53:57 on dist-test-slave-wg14"
I20260620 01:54:13.858143 7227 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:13.889364 7227 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:13.898530 7227 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:13.898896 7227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:13.928448 7227 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44547
I20260620 01:54:13.928505 7297 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44547 every 8 connection(s)
I20260620 01:54:13.931262 7227 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:54:13.938664 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 7227
I20260620 01:54:13.939082 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 6784
I20260620 01:54:13.943645 7298 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:54:13.948863 7298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: Bootstrap starting.
Jun 20 01:54:13 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 01:54:13.969316 6920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:54:13.974190 7298 log.cc:826] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: Log is configured to *not* fsync() on all Append() calls
I20260620 01:54:13.988121 7298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:54:13.988843 7298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: Bootstrap complete.
I20260620 01:54:13.998761 7298 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:54:14.000006 7298 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4345ecdc1b104a6096e8744d2614ec04, State: Initialized, Role: FOLLOWER
I20260620 01:54:14.001340 7298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:54:14.002610 7304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:14.003098 7304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:14.004091 7298 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:54:14.019369 7316 catalog_manager.cc:1294] Loaded cluster ID: 406167b0b78a4a00b6eb7f76168e5158
I20260620 01:54:14.019646 7316 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: loading cluster ID for follower catalog manager: success
I20260620 01:54:14.027778 7316 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: acquiring CA information for follower catalog manager: success
I20260620 01:54:14.035326 7316 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:54:14.091198 7227 master_runner.cc:264] No leader master found from master 4345ecdc1b104a6096e8744d2614ec04
I20260620 01:54:14.091447 7227 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:54:14.098610 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 6784
I20260620 01:54:14.099519 7217 heartbeater.cc:499] Master 127.20.1.190:44547 was elected leader, sending a full tablet report...
I20260620 01:54:14.113911 7261 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "845c0605282142148150cc4c3dd75d66" instance_seqno: 1781920453025840) as {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:35165; Asking this server to re-register.
I20260620 01:54:14.115550 7217 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:14.118330 7261 ts_manager.cc:194] Registered new tserver with Master: 845c0605282142148150cc4c3dd75d66 (127.20.1.131:45221)
I20260620 01:54:14.121690 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35207
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--webserver_port=44777
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:14.360150 7073 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44547
I20260620 01:54:14.360806 7073 heartbeater.cc:499] Master 127.20.1.190:44547 was elected leader, sending a full tablet report...
I20260620 01:54:14.362767 7261 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c669bcb3ef864811af0150ff4f216757" instance_seqno: 1781920452251672) as {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:57893; Asking this server to re-register.
I20260620 01:54:14.364079 7073 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:14.366223 7261 ts_manager.cc:194] Registered new tserver with Master: c669bcb3ef864811af0150ff4f216757 (127.20.1.130:40053)
I20260620 01:54:14.403648 7325 raft_consensus.cc:493] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4345ecdc1b104a6096e8744d2614ec04)
I20260620 01:54:14.403954 7325 raft_consensus.cc:515] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:54:14.414557 7325 leader_election.cc:290] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4345ecdc1b104a6096e8744d2614ec04 (127.20.1.190:44547), 9058f04895554d8585c606f129db9310 (127.20.1.188:44059)
I20260620 01:54:14.429394 7273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd052da5773244dd80b5fbbd78139b44" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4345ecdc1b104a6096e8744d2614ec04" is_pre_election: true
I20260620 01:54:14.430254 7273 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd052da5773244dd80b5fbbd78139b44 in term 2.
I20260620 01:54:14.431474 6612 leader_election.cc:304] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4345ecdc1b104a6096e8744d2614ec04, cd052da5773244dd80b5fbbd78139b44; no voters:
I20260620 01:54:14.432096 7325 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 01:54:14.432384 7325 raft_consensus.cc:493] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4345ecdc1b104a6096e8744d2614ec04)
I20260620 01:54:14.432542 7325 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:54:14.432215 6726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd052da5773244dd80b5fbbd78139b44" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9058f04895554d8585c606f129db9310" is_pre_election: true
I20260620 01:54:14.436218 7325 raft_consensus.cc:515] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:54:14.437234 7325 leader_election.cc:290] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [CANDIDATE]: Term 3 election: Requested vote from peers 4345ecdc1b104a6096e8744d2614ec04 (127.20.1.190:44547), 9058f04895554d8585c606f129db9310 (127.20.1.188:44059)
I20260620 01:54:14.438614 7273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd052da5773244dd80b5fbbd78139b44" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4345ecdc1b104a6096e8744d2614ec04"
I20260620 01:54:14.438951 7273 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:54:14.439783 6726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd052da5773244dd80b5fbbd78139b44" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9058f04895554d8585c606f129db9310"
I20260620 01:54:14.446687 7273 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd052da5773244dd80b5fbbd78139b44 in term 3.
I20260620 01:54:14.447687 6612 leader_election.cc:304] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4345ecdc1b104a6096e8744d2614ec04, cd052da5773244dd80b5fbbd78139b44; no voters: 9058f04895554d8585c606f129db9310
I20260620 01:54:14.448362 7325 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 01:54:14.449095 7325 raft_consensus.cc:697] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [term 3 LEADER]: Becoming Leader. State: Replica: cd052da5773244dd80b5fbbd78139b44, State: Running, Role: LEADER
I20260620 01:54:14.449826 7325 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:54:14.452302 7328 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd052da5773244dd80b5fbbd78139b44. Latest consensus state: current_term: 3 leader_uuid: "cd052da5773244dd80b5fbbd78139b44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:14.452900 7328 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: This master's current role is: LEADER
I20260620 01:54:14.454025 7330 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:54:14.457628 7330 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:54:14.459681 7330 catalog_manager.cc:1294] Loaded cluster ID: 406167b0b78a4a00b6eb7f76168e5158
I20260620 01:54:14.459867 7330 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:54:14.462358 7330 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:54:14.464138 7330 catalog_manager.cc:6080] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44: Loaded TSK: 0
I20260620 01:54:14.465801 7330 catalog_manager.cc:1549] Initializing in-progress tserver states...
W20260620 01:54:14.653303 7323 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:14.654064 7323 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:14.654266 7323 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:14.666142 7323 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:14.666451 7323 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:54:14.683367 7323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=44777
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:14.685372 7323 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:14.687186 7323 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:14.705435 7335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:14.705308 7338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:14.705261 7336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:14.705636 7323 server_base.cc:1061] running on GCE node
I20260620 01:54:14.707306 7323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:14.710602 7323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:14.712217 7323 hybrid_clock.cc:648] HybridClock initialized: now 1781920454712121 us; error 93 us; skew 500 ppm
I20260620 01:54:14.712809 7323 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:14 dist-test-slave-wg14 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920454, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:14.719883 7323 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:54:14.723162 7323 webserver.cc:533] Webserver started at http://127.20.1.129:44777/ using document root <none> and password file <none>
I20260620 01:54:14.724220 7323 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:14.724356 7323 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:14.732307 7323 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.007s
I20260620 01:54:14.739574 7345 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:14.741887 7323 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260620 01:54:14.742153 7323 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "d0b25c1ac6d3440b986814453a81522c"
format_stamp: "Formatted at 2026-06-20 01:54:00 on dist-test-slave-wg14"
I20260620 01:54:14.743080 7323 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:14.770030 7350 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4345ecdc1b104a6096e8744d2614ec04)
I20260620 01:54:14.770349 7350 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } }
I20260620 01:54:14.772680 7350 leader_election.cc:290] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4345ecdc1b104a6096e8744d2614ec04 (127.20.1.190:44547), cd052da5773244dd80b5fbbd78139b44 (127.20.1.189:34883)
I20260620 01:54:14.773984 6635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9058f04895554d8585c606f129db9310" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd052da5773244dd80b5fbbd78139b44" is_pre_election: true
I20260620 01:54:14.782217 7273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9058f04895554d8585c606f129db9310" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4345ecdc1b104a6096e8744d2614ec04" is_pre_election: true
I20260620 01:54:14.782747 7273 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9058f04895554d8585c606f129db9310 in current term 3: Already voted for candidate cd052da5773244dd80b5fbbd78139b44 in this term.
I20260620 01:54:14.782909 7323 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:14.783874 6703 leader_election.cc:304] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9058f04895554d8585c606f129db9310; no voters: 4345ecdc1b104a6096e8744d2614ec04, cd052da5773244dd80b5fbbd78139b44
I20260620 01:54:14.784461 7350 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:54:14.787798 7350 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 01:54:14.790484 7323 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:14.791597 7323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:14.793869 7323 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:14.796202 7323 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:14.796384 7323 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:14.796494 7323 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:14.796583 7323 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:14.846396 7323 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35207
I20260620 01:54:14.846640 7460 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35207 every 8 connection(s)
I20260620 01:54:14.848600 7323 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:54:14.849738 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 7323
Jun 20 01:54:14 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920454, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:54:14 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920454, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:54:14 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920454, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:54:14.906422 7462 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44547
I20260620 01:54:14.907068 7462 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:14.908507 7462 heartbeater.cc:507] Master 127.20.1.190:44547 requested a full tablet report, sending...
I20260620 01:54:14.909827 7461 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44059
I20260620 01:54:14.910049 7461 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:14.911067 7461 heartbeater.cc:507] Master 127.20.1.188:44059 requested a full tablet report, sending...
I20260620 01:54:14.911325 7261 ts_manager.cc:194] Registered new tserver with Master: d0b25c1ac6d3440b986814453a81522c (127.20.1.129:35207)
I20260620 01:54:14.913429 7463 heartbeater.cc:344] Connected to a master server at 127.20.1.189:34883
I20260620 01:54:14.913429 6715 ts_manager.cc:194] Re-registered known tserver with Master: d0b25c1ac6d3440b986814453a81522c (127.20.1.129:35207)
I20260620 01:54:14.913668 7463 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:14.914381 7463 heartbeater.cc:507] Master 127.20.1.189:34883 requested a full tablet report, sending...
I20260620 01:54:14.916110 6625 ts_manager.cc:194] Re-registered known tserver with Master: d0b25c1ac6d3440b986814453a81522c (127.20.1.129:35207)
I20260620 01:54:14.919188 6625 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:42871
I20260620 01:54:14.920200 7273 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [term 3 FOLLOWER]: Refusing update from remote peer cd052da5773244dd80b5fbbd78139b44: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:54:14.921789 7328 consensus_queue.cc:1048] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:54:14.927886 7474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd052da5773244dd80b5fbbd78139b44. Latest consensus state: current_term: 3 leader_uuid: "cd052da5773244dd80b5fbbd78139b44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:14.928326 7474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:14.930042 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 7323
W20260620 01:54:14.930059 7473 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:14.919052 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.20.1.129:35207)
0620 01:54:14.926303 (+ 7251us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:14.926312 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:14.926352 (+ 40us) server_negotiation.cc:408] Connection header received
0620 01:54:14.926429 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:14.926436 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:14.926686 (+ 250us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:14.926951 (+ 265us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:14.928409 (+ 1458us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60544: BlockingRecv error: recv got EOF from 127.0.0.1:60544 (error 108)
Metrics: {"server-negotiator.queue_time_us":7037,"thread_start_us":6788,"threads_started":1}
W20260620 01:54:14.932881 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 7323: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:35207: server requires authentication, but client does not have Kerberos credentials available
W20260620 01:54:14.933286 7473 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:14.930906 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.20.1.129:35207)
0620 01:54:14.931314 (+ 408us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:14.931320 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:14.931345 (+ 25us) server_negotiation.cc:408] Connection header received
0620 01:54:14.931464 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:14.931470 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:14.931572 (+ 102us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:14.932623 (+ 1051us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:14.932989 (+ 366us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60556: BlockingRecv error: recv got EOF from 127.0.0.1:60556 (error 108)
Metrics: {"server-negotiator.queue_time_us":289}
I20260620 01:54:14.933615 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 7323
I20260620 01:54:14.952240 7328 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "cd052da5773244dd80b5fbbd78139b44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:14.952571 7328 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: This master's current role is: LEADER
I20260620 01:54:14.953193 6726 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [term 3 FOLLOWER]: Refusing update from remote peer cd052da5773244dd80b5fbbd78139b44: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:54:14.953882 7474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "cd052da5773244dd80b5fbbd78139b44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:14.954113 7474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4345ecdc1b104a6096e8744d2614ec04 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:14.954414 7325 consensus_queue.cc:1048] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:54:14.958151 7481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "cd052da5773244dd80b5fbbd78139b44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:14.958491 7350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd052da5773244dd80b5fbbd78139b44. Latest consensus state: current_term: 3 leader_uuid: "cd052da5773244dd80b5fbbd78139b44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:14.958997 7481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:14.959002 7350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9058f04895554d8585c606f129db9310 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:14.963181 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35207
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--webserver_port=44777
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:54:15.450395 7482 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:15.450803 7482 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:15.450951 7482 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:54:15.451053 7482 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:15.462697 7482 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:15.463032 7482 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:54:15.477088 7482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=44777
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:15.479233 7482 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:15.479566 7325 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "cd052da5773244dd80b5fbbd78139b44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4345ecdc1b104a6096e8744d2614ec04" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44547 } } peers { permanent_uuid: "cd052da5773244dd80b5fbbd78139b44" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34883 } } peers { permanent_uuid: "9058f04895554d8585c606f129db9310" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44059 } } }
I20260620 01:54:15.480171 7325 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd052da5773244dd80b5fbbd78139b44 [sys.catalog]: This master's current role is: LEADER
I20260620 01:54:15.481729 7482 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:15.495313 7491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:15.495777 7490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:15.495849 7493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:15.496945 7482 server_base.cc:1061] running on GCE node
I20260620 01:54:15.497764 7482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:15.499080 7482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:15.500602 7482 hybrid_clock.cc:648] HybridClock initialized: now 1781920455500477 us; error 105 us; skew 500 ppm
I20260620 01:54:15.501049 7482 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:15 dist-test-slave-wg14 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920455, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:15.506598 7482 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:54:15.510054 7482 webserver.cc:533] Webserver started at http://127.20.1.129:44777/ using document root <none> and password file <none>
I20260620 01:54:15.511226 7482 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:15.511379 7482 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:15.519285 7482 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 01:54:15.524250 7500 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:15.527081 7482 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.002s
I20260620 01:54:15.527459 7482 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "d0b25c1ac6d3440b986814453a81522c"
format_stamp: "Formatted at 2026-06-20 01:54:00 on dist-test-slave-wg14"
I20260620 01:54:15.528515 7482 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:15.549851 7482 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:15.557243 7482 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:15.557628 7482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:15.559333 7482 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:15.561794 7482 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:15.561911 7482 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:15.562024 7482 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:15.562077 7482 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:54:15.562829 7482 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:54:15.563000 7482 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:54:15.563112 7482 fault_injection.cc:59] injecting fault for kudu (pid 7482): FLAGS_fault_before_start (process will exit)
I20260620 01:54:15.579418 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35207
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--webserver_port=44777
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:16.102104 7218 heartbeater.cc:499] Master 127.20.1.189:34883 was elected leader, sending a full tablet report...
W20260620 01:54:16.128521 7508 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:16.129247 7508 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:16.129459 7508 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:16.141314 7508 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:16.141659 7508 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:54:16.158138 7508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=44777
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:44547,127.20.1.189:34883,127.20.1.188:44059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:16.160884 7508 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:16.162581 7508 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:16.176651 7514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:16.176920 7513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:16.178468 7516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:16.179464 7508 server_base.cc:1061] running on GCE node
I20260620 01:54:16.180809 7508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:16.182513 7508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:16.183990 7508 hybrid_clock.cc:648] HybridClock initialized: now 1781920456183907 us; error 67 us; skew 500 ppm
I20260620 01:54:16.184648 7508 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:16 dist-test-slave-wg14 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920456, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:16.189638 7508 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:54:16.193064 7508 webserver.cc:533] Webserver started at http://127.20.1.129:44777/ using document root <none> and password file <none>
I20260620 01:54:16.194850 7508 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:16.195071 7508 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:16.204525 7508 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:54:16.209954 7523 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:16.212797 7508 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.004s sys 0.001s
I20260620 01:54:16.213150 7508 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "d0b25c1ac6d3440b986814453a81522c"
format_stamp: "Formatted at 2026-06-20 01:54:00 on dist-test-slave-wg14"
I20260620 01:54:16.214326 7508 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:16.258204 7508 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:16.265470 7508 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:16.265926 7508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:16.267485 7508 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:16.270130 7508 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:16.270280 7508 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:16.270403 7508 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:16.270483 7508 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:16.325189 7508 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35207
I20260620 01:54:16.325255 7636 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35207 every 8 connection(s)
I20260620 01:54:16.327265 7508 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:54:16.335708 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 7508
I20260620 01:54:16.336162 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 7508
I20260620 01:54:16.355410 7074 heartbeater.cc:499] Master 127.20.1.189:34883 was elected leader, sending a full tablet report...
W20260620 01:54:16.358824 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 7508: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:35207: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:54:16.359083 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 7508
Jun 20 01:54:16 dist-test-slave-wg14 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920456, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:54:16.377877 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 6939
W20260620 01:54:16.379243 7641 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:16.332496 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:35813 (local address 127.20.1.188:44059)
0620 01:54:16.333466 (+ 970us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:16.333476 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:16.336364 (+ 2888us) server_negotiation.cc:408] Connection header received
0620 01:54:16.340711 (+ 4347us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:16.340720 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:16.340846 (+ 126us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:16.340952 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:16.345403 (+ 4451us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:54:16.347260 (+ 1857us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:54:16.355939 (+ 8679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:54:16.356831 (+ 892us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0620 01:54:16.377782 (+ 20951us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:35813: BlockingRecv error: recv got EOF from 127.20.1.129:35813 (error 108)
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":455,"threads_started":1}
W20260620 01:54:16.379302 7644 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:16.334656 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:39263 (local address 127.20.1.190:44547)
0620 01:54:16.341265 (+ 6609us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:16.341274 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:16.341307 (+ 33us) server_negotiation.cc:408] Connection header received
0620 01:54:16.341396 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:16.341404 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:16.341508 (+ 104us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:16.341711 (+ 203us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:16.354310 (+ 12599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:54:16.356020 (+ 1710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:54:16.377833 (+ 21813us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:39263: BlockingRecv error: recv got EOF from 127.20.1.129:39263 (error 108)
Metrics: {"server-negotiator.queue_time_us":6422,"thread_start_us":6254,"threads_started":1}
W20260620 01:54:16.379308 7647 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:16.337695 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:54021 (local address 127.20.1.189:34883)
0620 01:54:16.348329 (+ 10634us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:16.348337 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:16.353967 (+ 5630us) server_negotiation.cc:408] Connection header received
0620 01:54:16.354063 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:16.354070 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:16.354204 (+ 134us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:16.354381 (+ 177us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:16.361088 (+ 6707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:54:16.362288 (+ 1200us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:54:16.377816 (+ 15528us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:54021: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":10435,"thread_start_us":10061,"threads_started":1}
W20260620 01:54:16.382545 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 6939: Not authorized: Client connection negotiation failed: client connection to 127.20.1.130:40053: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:54:16.382850 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 6939
I20260620 01:54:16.399641 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 7087
W20260620 01:54:16.403687 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 7087: Not authorized: Client connection negotiation failed: client connection to 127.20.1.131:45221: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:54:16.403851 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 7087
I20260620 01:54:16.422508 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 7227
W20260620 01:54:16.426713 7644 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:16.424664 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:57014 (local address 127.20.1.190:44547)
0620 01:54:16.424962 (+ 298us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:16.424969 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:16.424994 (+ 25us) server_negotiation.cc:408] Connection header received
0620 01:54:16.425068 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:16.425073 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:16.425155 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:16.425286 (+ 131us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:16.426503 (+ 1217us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57014: BlockingRecv error: recv got EOF from 127.0.0.1:57014 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
W20260620 01:54:16.426620 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 7227: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:44547: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:54:16.427078 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 7227
I20260620 01:54:16.450922 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 6593
I20260620 01:54:16.453049 6655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:54:16.598214 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 6593
W20260620 01:54:16.620726 6703 connection.cc:573] server connection from 127.0.0.1:44042 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:54:16.621235 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 6683
I20260620 01:54:16.622920 6746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:54:16.747531 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 6683
2026-06-20T01:54:16Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/7 (22175 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/8
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 01:54:16 dist-test-slave-wg14 krb5kdc[7655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 01:54:16 dist-test-slave-wg14 krb5kdc[7655](info): set up 2 sockets
Jun 20 01:54:16 dist-test-slave-wg14 krb5kdc[7655](info): commencing operation
krb5kdc: starting...
W20260620 01:54:18.856210 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.059s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 01:54:19 dist-test-slave-wg14 krb5kdc[7655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T01:54:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:54:19Z Disabled control of system clock
2026-06-20T01:54:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:54:19Z Disabled control of system clock
2026-06-20T01:54:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:54:19Z Disabled control of system clock
2026-06-20T01:54:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:54:19Z Disabled control of system clock
2026-06-20T01:54:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:54:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:19.409096 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42071
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:54:19.901610 7679 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:19.902015 7679 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:19.902133 7679 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:19.915208 7679 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:54:19.915396 7679 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:19.915509 7679 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:54:19.915585 7679 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:54:19.937328 7679 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:19.939931 7679 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:19.941955 7679 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:19.955480 7685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:19.955542 7684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:19.957942 7687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:19.963276 7679 server_base.cc:1061] running on GCE node
I20260620 01:54:19.965452 7679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:19.968880 7679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:19.972767 7679 hybrid_clock.cc:648] HybridClock initialized: now 1781920459972387 us; error 216 us; skew 500 ppm
I20260620 01:54:19.973560 7679 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:19 dist-test-slave-wg14 krb5kdc[7655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920459, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:19.979984 7679 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:54:19.983911 7679 webserver.cc:533] Webserver started at http://127.20.1.190:34645/ using document root <none> and password file <none>
I20260620 01:54:19.986457 7679 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:19.986792 7679 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:19.987764 7679 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:54:19.992066 7679 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "f618f87e8d484910a4586badc3211eb0"
format_stamp: "Formatted at 2026-06-20 01:54:19 on dist-test-slave-wg14"
I20260620 01:54:19.993530 7679 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "f618f87e8d484910a4586badc3211eb0"
format_stamp: "Formatted at 2026-06-20 01:54:19 on dist-test-slave-wg14"
I20260620 01:54:20.003455 7679 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.011s sys 0.000s
I20260620 01:54:20.009469 7694 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:20.011740 7679 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260620 01:54:20.012091 7679 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "f618f87e8d484910a4586badc3211eb0"
format_stamp: "Formatted at 2026-06-20 01:54:19 on dist-test-slave-wg14"
I20260620 01:54:20.012391 7679 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:20.055804 7679 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:20.069453 7679 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:20.070001 7679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:20.113770 7679 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:42071
I20260620 01:54:20.113816 7745 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:42071 every 8 connection(s)
I20260620 01:54:20.115862 7679 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:54:20.117368 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 7679
I20260620 01:54:20.117615 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:54:20.123620 7746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:54:20.135344 7746 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.189@KRBTEST.COM" created.
Jun 20 01:54:20 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920459, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:54:20.176175 7746 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:54:20.177812 7697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:37363: connect: Connection refused (error 111)
W20260620 01:54:20.180111 7746 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:37363: Network error: Client connection negotiation failed: client connection to 127.20.1.189:37363: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:20.230016 7746 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } attempt: 1
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
W20260620 01:54:20.233896 7746 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:37363: Network error: Client connection negotiation failed: client connection to 127.20.1.189:37363: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:20.281947 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:37363
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:20.302560 7746 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } attempt: 2
W20260620 01:54:20.306658 7746 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:37363: Network error: Client connection negotiation failed: client connection to 127.20.1.189:37363: connect: Connection refused (error 111)
I20260620 01:54:20.398341 7746 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } attempt: 3
W20260620 01:54:20.402408 7746 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:37363: Network error: Client connection negotiation failed: client connection to 127.20.1.189:37363: connect: Connection refused (error 111)
I20260620 01:54:20.545831 7746 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } attempt: 4
W20260620 01:54:20.548951 7746 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:37363: Network error: Client connection negotiation failed: client connection to 127.20.1.189:37363: connect: Connection refused (error 111)
W20260620 01:54:20.749948 7756 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:20.750389 7756 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:20.750545 7756 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:20.761690 7756 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:54:20.761893 7756 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:20.761988 7756 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:54:20.762065 7756 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:54:20.775889 7756 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:37363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:20.778319 7756 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:20.780435 7756 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:20.793462 7764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:20.793623 7761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:20.793491 7762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:20.794010 7756 server_base.cc:1061] running on GCE node
I20260620 01:54:20.795461 7756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:20.797824 7756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:20.801504 7756 hybrid_clock.cc:648] HybridClock initialized: now 1781920460801279 us; error 121 us; skew 500 ppm
I20260620 01:54:20.802098 7756 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:20 dist-test-slave-wg14 krb5kdc[7655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920460, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:20.806718 7756 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 01:54:20.809561 7756 webserver.cc:533] Webserver started at http://127.20.1.189:45637/ using document root <none> and password file <none>
I20260620 01:54:20.810694 7756 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:20.810873 7756 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:20.811400 7756 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:54:20.814574 7756 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "9322a24f7def435a92159114e88b9d7c"
format_stamp: "Formatted at 2026-06-20 01:54:20 on dist-test-slave-wg14"
I20260620 01:54:20.815658 7756 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "9322a24f7def435a92159114e88b9d7c"
format_stamp: "Formatted at 2026-06-20 01:54:20 on dist-test-slave-wg14"
I20260620 01:54:20.824491 7756 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260620 01:54:20.830441 7771 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:20.832890 7756 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260620 01:54:20.833194 7756 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "9322a24f7def435a92159114e88b9d7c"
format_stamp: "Formatted at 2026-06-20 01:54:20 on dist-test-slave-wg14"
I20260620 01:54:20.833464 7756 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:20.848358 7746 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } attempt: 5
W20260620 01:54:20.851598 7746 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:37363: Network error: Client connection negotiation failed: client connection to 127.20.1.189:37363: connect: Connection refused (error 111)
I20260620 01:54:20.865281 7756 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:20.870160 7756 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:20.870502 7756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:20.896350 7756 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:37363
I20260620 01:54:20.896441 7822 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:37363 every 8 connection(s)
I20260620 01:54:20.898347 7756 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:54:20.903755 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 7756
I20260620 01:54:20.903988 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:54:20.904511 7823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:54:20.912073 7823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:54:20 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920460, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.188@KRBTEST.COM" created.
I20260620 01:54:20.945140 7823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:54:20 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920460, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:54:20.970139 7823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:54:20.972267 7774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46173: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:54:20.975076 7823 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46173: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46173: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.188@KRBTEST.COM" created.
I20260620 01:54:21.024598 7823 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } attempt: 1
W20260620 01:54:21.027957 7823 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46173: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46173: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:21.040882 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:46173
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:21.096391 7823 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } attempt: 2
W20260620 01:54:21.099252 7823 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46173: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46173: connect: Connection refused (error 111)
I20260620 01:54:21.190574 7823 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } attempt: 3
W20260620 01:54:21.193996 7823 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46173: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46173: connect: Connection refused (error 111)
I20260620 01:54:21.337303 7823 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } attempt: 4
W20260620 01:54:21.340574 7823 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46173: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46173: connect: Connection refused (error 111)
I20260620 01:54:21.398926 7746 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } attempt: 6
Jun 20 01:54:21 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920459, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:54:21.419693 7746 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:54:21.423309 7746 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46173: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46173: connect: Connection refused (error 111)
I20260620 01:54:21.475682 7746 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } attempt: 1
W20260620 01:54:21.479359 7746 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46173: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46173: connect: Connection refused (error 111)
W20260620 01:54:21.494170 7833 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:21.494671 7833 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:21.494813 7833 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:21.504505 7833 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:54:21.504700 7833 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:21.504783 7833 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:54:21.504853 7833 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:54:21.517447 7833 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:46173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:21.519506 7833 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:21.521198 7833 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:21.534301 7841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:21.534264 7840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:21.535032 7843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:21.537313 7833 server_base.cc:1061] running on GCE node
I20260620 01:54:21.538728 7833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:21.541368 7833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:21.545146 7833 hybrid_clock.cc:648] HybridClock initialized: now 1781920461544847 us; error 109 us; skew 500 ppm
I20260620 01:54:21.545840 7833 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:21 dist-test-slave-wg14 krb5kdc[7655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920461, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:21.551667 7833 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 01:54:21.553694 7746 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } attempt: 2
I20260620 01:54:21.554926 7833 webserver.cc:533] Webserver started at http://127.20.1.188:34381/ using document root <none> and password file <none>
I20260620 01:54:21.556229 7833 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:21.556438 7833 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260620 01:54:21.556559 7746 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46173: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46173: connect: Connection refused (error 111)
I20260620 01:54:21.556896 7833 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:54:21.560320 7833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "6edf4a403b194538857d38331fb4c16d"
format_stamp: "Formatted at 2026-06-20 01:54:21 on dist-test-slave-wg14"
I20260620 01:54:21.561479 7833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "6edf4a403b194538857d38331fb4c16d"
format_stamp: "Formatted at 2026-06-20 01:54:21 on dist-test-slave-wg14"
I20260620 01:54:21.570855 7833 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20260620 01:54:21.576350 7850 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:21.578747 7833 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:54:21.579051 7833 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "6edf4a403b194538857d38331fb4c16d"
format_stamp: "Formatted at 2026-06-20 01:54:21 on dist-test-slave-wg14"
I20260620 01:54:21.579409 7833 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:21.632855 7833 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:21.639907 7823 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } attempt: 5
I20260620 01:54:21.640358 7833 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:21.640744 7833 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260620 01:54:21.643677 7823 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46173: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46173: connect: Connection refused (error 111)
I20260620 01:54:21.669885 7746 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } attempt: 3
W20260620 01:54:21.673588 7746 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46173: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46173: connect: Connection refused (error 111)
I20260620 01:54:21.673913 7902 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:46173 every 8 connection(s)
I20260620 01:54:21.673923 7833 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:46173
I20260620 01:54:21.676687 7833 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:54:21.682951 7903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:54:21.685324 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 7833
I20260620 01:54:21.685565 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:54:21.694669 7903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:54:21 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:54:21 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920461, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:54:21 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:54:21.726270 7903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:54:21 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920461, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:54:21 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:54:21.751824 7903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:54:21 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920461, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:54:21.778710 7903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: Bootstrap starting.
I20260620 01:54:21.782614 7903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: Neither blocks nor log segments found. Creating new log.
I20260620 01:54:21.783924 7903 log.cc:826] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: Log is configured to *not* fsync() on all Append() calls
I20260620 01:54:21.788054 7903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: No bootstrap required, opened a new log
I20260620 01:54:21.795025 7903 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:21.795885 7903 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:54:21.796064 7903 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edf4a403b194538857d38331fb4c16d, State: Initialized, Role: FOLLOWER
I20260620 01:54:21.796849 7903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:21.798336 7913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:21.798657 7913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:21.800077 7903 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:54:21.813815 7924 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:54:21.814039 7924 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:54:21.810681 7833 master_runner.cc:187] Error getting master registration for 127.20.1.190:42071: OK, instance_id { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" instance_seqno: 1781920460098477 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:54:21.820176 7833 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:54:21.822944 7746 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } attempt: 4
Jun 20 01:54:21 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920459, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:54:21.851984 7746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: Bootstrap starting.
I20260620 01:54:21.856048 7746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: Neither blocks nor log segments found. Creating new log.
I20260620 01:54:21.857722 7746 log.cc:826] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: Log is configured to *not* fsync() on all Append() calls
I20260620 01:54:21.861899 7746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: No bootstrap required, opened a new log
I20260620 01:54:21.869643 7746 raft_consensus.cc:359] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:21.870111 7746 raft_consensus.cc:385] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:54:21.870278 7746 raft_consensus.cc:740] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f618f87e8d484910a4586badc3211eb0, State: Initialized, Role: FOLLOWER
I20260620 01:54:21.871225 7746 consensus_queue.cc:260] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:21.873082 7927 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:21.873564 7927 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:21.874691 7746 sys_catalog.cc:565] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:54:21.888863 7938 catalog_manager.cc:1593] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:54:21.889099 7938 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:54:21.891232 7927 raft_consensus.cc:493] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:54:21.891618 7927 raft_consensus.cc:515] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
W20260620 01:54:21.894109 7856 tablet.cc:2406] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:54:21.889643 7679 master_runner.cc:187] Error getting master registration for 127.20.1.189:37363: OK, instance_id { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" instance_seqno: 1781920460888706 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:54:21.899828 7927 leader_election.cc:290] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9322a24f7def435a92159114e88b9d7c (127.20.1.189:37363), 6edf4a403b194538857d38331fb4c16d (127.20.1.188:46173)
I20260620 01:54:21.900386 7679 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:54:21.901729 7877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f618f87e8d484910a4586badc3211eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edf4a403b194538857d38331fb4c16d" is_pre_election: true
I20260620 01:54:21.902609 7877 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f618f87e8d484910a4586badc3211eb0 in term 0.
I20260620 01:54:21.905109 7697 leader_election.cc:304] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edf4a403b194538857d38331fb4c16d, f618f87e8d484910a4586badc3211eb0; no voters:
I20260620 01:54:21.905778 7927 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:54:21.906018 7927 raft_consensus.cc:493] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:54:21.906140 7927 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:54:21.910499 7927 raft_consensus.cc:515] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:21.914455 7877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f618f87e8d484910a4586badc3211eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edf4a403b194538857d38331fb4c16d"
I20260620 01:54:21.914815 7877 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:54:21.919697 7927 leader_election.cc:290] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 9322a24f7def435a92159114e88b9d7c (127.20.1.189:37363), 6edf4a403b194538857d38331fb4c16d (127.20.1.188:46173)
I20260620 01:54:21.923516 7877 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f618f87e8d484910a4586badc3211eb0 in term 1.
I20260620 01:54:21.926834 7697 leader_election.cc:304] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edf4a403b194538857d38331fb4c16d, f618f87e8d484910a4586badc3211eb0; no voters:
I20260620 01:54:21.927461 7927 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:54:21.928206 7927 raft_consensus.cc:697] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 1 LEADER]: Becoming Leader. State: Replica: f618f87e8d484910a4586badc3211eb0, State: Running, Role: LEADER
I20260620 01:54:21.929430 7927 consensus_queue.cc:237] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:21.936664 7940 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f618f87e8d484910a4586badc3211eb0. Latest consensus state: current_term: 1 leader_uuid: "f618f87e8d484910a4586badc3211eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:21.937243 7940 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: This master's current role is: LEADER
I20260620 01:54:21.944352 7942 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:54:21.947680 7942 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:54:21.968514 7877 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 1 FOLLOWER]: Refusing update from remote peer f618f87e8d484910a4586badc3211eb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:54:21.970281 7940 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:54:21.994390 7913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f618f87e8d484910a4586badc3211eb0. Latest consensus state: current_term: 1 leader_uuid: "f618f87e8d484910a4586badc3211eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:21.994781 7913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:22.022953 7913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f618f87e8d484910a4586badc3211eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:22.023367 7913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:22.026216 7940 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f618f87e8d484910a4586badc3211eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:22.026572 7940 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: This master's current role is: LEADER
I20260620 01:54:22.033612 7942 catalog_manager.cc:1382] Generated new cluster ID: 7c89cc2f428d4a57adaab580812f43ad
I20260620 01:54:22.033849 7942 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:54:22.063498 7942 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:54:22.067497 7942 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:54:22.112885 7942 catalog_manager.cc:6069] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: Generated new TSK 0
I20260620 01:54:22.115731 7942 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:54:22.191022 7823 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } attempt: 6
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Jun 20 01:54:22 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920460, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:22.246131 7823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c: Bootstrap starting.
W20260620 01:54:22.247080 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.104s user 0.000s sys 0.024s
I20260620 01:54:22.259544 7823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c: Neither blocks nor log segments found. Creating new log.
I20260620 01:54:22.264464 7823 log.cc:826] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c: Log is configured to *not* fsync() on all Append() calls
I20260620 01:54:22.284153 7823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c: No bootstrap required, opened a new log
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
I20260620 01:54:22.307598 7823 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:22.308697 7823 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:54:22.309063 7823 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9322a24f7def435a92159114e88b9d7c, State: Initialized, Role: FOLLOWER
I20260620 01:54:22.311805 7823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:22.328634 7954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:22.329362 7954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:22.329708 7796 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:54:22.331372 7798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f618f87e8d484910a4586badc3211eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9322a24f7def435a92159114e88b9d7c" is_pre_election: true
I20260620 01:54:22.328778 7797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f618f87e8d484910a4586badc3211eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9322a24f7def435a92159114e88b9d7c"
I20260620 01:54:22.334059 7823 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:54:22.335412 7796 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 1 FOLLOWER]: Refusing update from remote peer f618f87e8d484910a4586badc3211eb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:54:22.337427 7927 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:22.346833 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:22.376012 7954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f618f87e8d484910a4586badc3211eb0. Latest consensus state: current_term: 1 leader_uuid: "f618f87e8d484910a4586badc3211eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:22.376950 7954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:22.377048 7964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f618f87e8d484910a4586badc3211eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:22.377435 7964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:22.467451 7971 catalog_manager.cc:1294] Loaded cluster ID: 7c89cc2f428d4a57adaab580812f43ad
I20260620 01:54:22.467579 7971 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c: loading cluster ID for follower catalog manager: success
I20260620 01:54:22.471035 7971 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c: acquiring CA information for follower catalog manager: success
I20260620 01:54:22.475075 7971 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:54:22.798817 7927 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f618f87e8d484910a4586badc3211eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:22.799542 7927 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: This master's current role is: LEADER
I20260620 01:54:22.818974 7924 catalog_manager.cc:1294] Loaded cluster ID: 7c89cc2f428d4a57adaab580812f43ad
I20260620 01:54:22.819330 7924 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: loading cluster ID for follower catalog manager: success
I20260620 01:54:22.825489 7924 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: acquiring CA information for follower catalog manager: success
I20260620 01:54:22.829459 7924 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 01:54:22.964457 7963 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:22.964887 7963 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:22.965039 7963 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:22.976200 7963 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:22.976599 7963 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:54:22.990001 7963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:22.992280 7963 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:22.994475 7963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:23.008147 7980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:23.010424 7981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:23.011582 7983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:23.014076 7963 server_base.cc:1061] running on GCE node
I20260620 01:54:23.015187 7963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:23.017226 7963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:23.021116 7963 hybrid_clock.cc:648] HybridClock initialized: now 1781920463020805 us; error 226 us; skew 500 ppm
I20260620 01:54:23.021721 7963 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:23 dist-test-slave-wg14 krb5kdc[7655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920463, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:23.027889 7963 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:54:23.030381 7963 webserver.cc:533] Webserver started at http://127.20.1.129:40753/ using document root <none> and password file <none>
I20260620 01:54:23.031567 7963 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:23.031764 7963 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:23.032176 7963 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:54:23.035099 7963 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "cbec150f8e594225b4d792c3284bb950"
format_stamp: "Formatted at 2026-06-20 01:54:23 on dist-test-slave-wg14"
I20260620 01:54:23.036233 7963 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "cbec150f8e594225b4d792c3284bb950"
format_stamp: "Formatted at 2026-06-20 01:54:23 on dist-test-slave-wg14"
I20260620 01:54:23.043853 7963 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260620 01:54:23.049464 7990 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:23.051812 7963 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:54:23.052088 7963 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "cbec150f8e594225b4d792c3284bb950"
format_stamp: "Formatted at 2026-06-20 01:54:23 on dist-test-slave-wg14"
I20260620 01:54:23.052359 7963 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:23.074115 7963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:23.080974 7963 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:23.081319 7963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:23.082619 7963 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:23.084905 7963 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:23.085041 7963 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:23.085194 7963 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:23.085259 7963 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:23.131956 7963 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:36051
I20260620 01:54:23.132021 8104 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:36051 every 8 connection(s)
I20260620 01:54:23.134232 7963 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:54:23.141214 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 7963
I20260620 01:54:23.141453 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 01:54:23 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920463, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:54:23 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920463, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:54:23 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920463, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:54:23.173791 8107 heartbeater.cc:344] Connected to a master server at 127.20.1.189:37363
I20260620 01:54:23.174376 8107 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:23.175673 8107 heartbeater.cc:507] Master 127.20.1.189:37363 requested a full tablet report, sending...
I20260620 01:54:23.177731 8105 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46173
I20260620 01:54:23.177942 8105 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:23.178715 8105 heartbeater.cc:507] Master 127.20.1.188:46173 requested a full tablet report, sending...
I20260620 01:54:23.179286 7788 ts_manager.cc:194] Registered new tserver with Master: cbec150f8e594225b4d792c3284bb950 (127.20.1.129:36051)
I20260620 01:54:23.180099 8106 heartbeater.cc:344] Connected to a master server at 127.20.1.190:42071
I20260620 01:54:23.180305 8106 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:23.180893 8106 heartbeater.cc:507] Master 127.20.1.190:42071 requested a full tablet report, sending...
I20260620 01:54:23.181412 7867 ts_manager.cc:194] Registered new tserver with Master: cbec150f8e594225b4d792c3284bb950 (127.20.1.129:36051)
WARNING: no policy specified for kudu/127.20.1.130@KRBTEST.COM; defaulting to no policy
I20260620 01:54:23.183029 7711 ts_manager.cc:194] Registered new tserver with Master: cbec150f8e594225b4d792c3284bb950 (127.20.1.129:36051)
I20260620 01:54:23.184938 7711 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:39001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:23.283679 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:54:23.721763 8119 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:23.722605 8119 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:23.722769 8119 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:23.732414 8119 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:23.732625 8119 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:54:23.747357 8119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:23.749300 8119 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:23.751102 8119 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:23.766194 8124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:23.766234 8125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:23.767458 8127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:23.768293 8119 server_base.cc:1061] running on GCE node
I20260620 01:54:23.769167 8119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:23.770884 8119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:23.774477 8119 hybrid_clock.cc:648] HybridClock initialized: now 1781920463774193 us; error 189 us; skew 500 ppm
I20260620 01:54:23.775049 8119 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:23 dist-test-slave-wg14 krb5kdc[7655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920463, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:23.779793 8119 init.cc:377] Logged in from keytab as kudu/127.20.1.130@KRBTEST.COM (short username kudu)
I20260620 01:54:23.782647 8119 webserver.cc:533] Webserver started at http://127.20.1.130:42947/ using document root <none> and password file <none>
I20260620 01:54:23.783843 8119 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:23.784013 8119 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:23.784485 8119 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:54:23.789539 8119 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/data/instance:
uuid: "67f8a95b2d2d4d3db088a4777111b11e"
format_stamp: "Formatted at 2026-06-20 01:54:23 on dist-test-slave-wg14"
I20260620 01:54:23.790892 8119 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance:
uuid: "67f8a95b2d2d4d3db088a4777111b11e"
format_stamp: "Formatted at 2026-06-20 01:54:23 on dist-test-slave-wg14"
I20260620 01:54:23.800348 8119 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.002s sys 0.005s
I20260620 01:54:23.807374 8134 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:23.810274 8119 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260620 01:54:23.810573 8119 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "67f8a95b2d2d4d3db088a4777111b11e"
format_stamp: "Formatted at 2026-06-20 01:54:23 on dist-test-slave-wg14"
I20260620 01:54:23.810853 8119 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:23.837882 8119 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:23.844713 8119 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:23.845041 8119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:23.846511 8119 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:23.848965 8119 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:23.849083 8119 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:23.849190 8119 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:23.849237 8119 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:23.909144 8119 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:43911
I20260620 01:54:23.909164 8248 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:43911 every 8 connection(s)
I20260620 01:54:23.911748 8119 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:54:23.912137 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 8119
I20260620 01:54:23.912384 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
Jun 20 01:54:23 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920463, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.131@KRBTEST.COM; defaulting to no policy
I20260620 01:54:23.973776 8250 heartbeater.cc:344] Connected to a master server at 127.20.1.190:42071
I20260620 01:54:23.974385 8250 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:23.975948 8250 heartbeater.cc:507] Master 127.20.1.190:42071 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.131@KRBTEST.COM" created.
I20260620 01:54:23.978214 7711 ts_manager.cc:194] Registered new tserver with Master: 67f8a95b2d2d4d3db088a4777111b11e (127.20.1.130:43911)
Jun 20 01:54:23 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920463, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:54:23 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920463, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:54:23.983103 7711 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:34631
I20260620 01:54:23.994215 8251 heartbeater.cc:344] Connected to a master server at 127.20.1.189:37363
I20260620 01:54:23.994491 8251 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:23.995643 8251 heartbeater.cc:507] Master 127.20.1.189:37363 requested a full tablet report, sending...
I20260620 01:54:23.997802 7788 ts_manager.cc:194] Registered new tserver with Master: 67f8a95b2d2d4d3db088a4777111b11e (127.20.1.130:43911)
I20260620 01:54:24.005554 8249 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46173
I20260620 01:54:24.005847 8249 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:24.006214 8249 heartbeater.cc:507] Master 127.20.1.188:46173 requested a full tablet report, sending...
I20260620 01:54:24.008760 7867 ts_manager.cc:194] Registered new tserver with Master: 67f8a95b2d2d4d3db088a4777111b11e (127.20.1.130:43911)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:54:24.033685 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.131: real 0.112s user 0.000s sys 0.027s
WARNING: no policy specified for HTTP/127.20.1.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:24.100382 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:24.190127 8106 heartbeater.cc:499] Master 127.20.1.190:42071 was elected leader, sending a full tablet report...
W20260620 01:54:24.563807 8263 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:24.564231 8263 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:24.564389 8263 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:24.575547 8263 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:24.575929 8263 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:54:24.590021 8263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:24.592018 8263 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:24.593748 8263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:24.607483 8269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:24.607467 8268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:24.609170 8271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:24.609633 8263 server_base.cc:1061] running on GCE node
I20260620 01:54:24.610461 8263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:24.612394 8263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:24.615976 8263 hybrid_clock.cc:648] HybridClock initialized: now 1781920464615748 us; error 126 us; skew 500 ppm
I20260620 01:54:24.616626 8263 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:24 dist-test-slave-wg14 krb5kdc[7655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920464, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:24.621994 8263 init.cc:377] Logged in from keytab as kudu/127.20.1.131@KRBTEST.COM (short username kudu)
I20260620 01:54:24.624883 8263 webserver.cc:533] Webserver started at http://127.20.1.131:37803/ using document root <none> and password file <none>
I20260620 01:54:24.626012 8263 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:24.626190 8263 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:24.626638 8263 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:54:24.629829 8263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/data/instance:
uuid: "483ef5d33e474f5290b962ec36c02a45"
format_stamp: "Formatted at 2026-06-20 01:54:24 on dist-test-slave-wg14"
I20260620 01:54:24.631044 8263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance:
uuid: "483ef5d33e474f5290b962ec36c02a45"
format_stamp: "Formatted at 2026-06-20 01:54:24 on dist-test-slave-wg14"
I20260620 01:54:24.639688 8263 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260620 01:54:24.644526 8278 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:24.646885 8263 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260620 01:54:24.647209 8263 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "483ef5d33e474f5290b962ec36c02a45"
format_stamp: "Formatted at 2026-06-20 01:54:24 on dist-test-slave-wg14"
I20260620 01:54:24.647472 8263 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:24.665127 8263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:24.671937 8263 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:24.672608 8263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:24.674734 8263 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:24.677757 8263 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:24.677922 8263 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:24.678068 8263 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:24.678133 8263 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:24.722618 8263 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:36341
I20260620 01:54:24.722694 8391 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:36341 every 8 connection(s)
I20260620 01:54:24.724603 8263 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:54:24.730862 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 8263
I20260620 01:54:24.731482 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
Jun 20 01:54:24 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920464, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:54:24 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920464, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:54:24 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920464, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:54:24.781919 8393 heartbeater.cc:344] Connected to a master server at 127.20.1.190:42071
I20260620 01:54:24.781951 8394 heartbeater.cc:344] Connected to a master server at 127.20.1.189:37363
I20260620 01:54:24.781911 8392 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46173
I20260620 01:54:24.782516 8393 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:24.782620 8392 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:24.782622 8394 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:24.784355 8393 heartbeater.cc:507] Master 127.20.1.190:42071 requested a full tablet report, sending...
I20260620 01:54:24.784449 8394 heartbeater.cc:507] Master 127.20.1.189:37363 requested a full tablet report, sending...
I20260620 01:54:24.784397 8392 heartbeater.cc:507] Master 127.20.1.188:46173 requested a full tablet report, sending...
I20260620 01:54:24.787828 7867 ts_manager.cc:194] Registered new tserver with Master: 483ef5d33e474f5290b962ec36c02a45 (127.20.1.131:36341)
I20260620 01:54:24.787818 7711 ts_manager.cc:194] Registered new tserver with Master: 483ef5d33e474f5290b962ec36c02a45 (127.20.1.131:36341)
I20260620 01:54:24.787911 7788 ts_manager.cc:194] Registered new tserver with Master: 483ef5d33e474f5290b962ec36c02a45 (127.20.1.131:36341)
I20260620 01:54:24.789325 7711 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:43657
I20260620 01:54:24.799487 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:54:24.811932 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:54:24.812170 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 7679
I20260620 01:54:24.813349 7741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:54:24.958894 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 7679
I20260620 01:54:24.987047 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42071
--webserver_interface=127.20.1.190
--webserver_port=34645
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:24.990993 8250 heartbeater.cc:499] Master 127.20.1.190:42071 was elected leader, sending a full tablet report...
W20260620 01:54:24.993973 8250 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:42071 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:42071: connect: Connection refused (error 111)
W20260620 01:54:25.195659 8106 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:42071 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:42071: connect: Connection refused (error 111)
W20260620 01:54:25.489614 8404 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:25.490011 8404 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:25.490113 8404 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:25.501811 8404 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:54:25.501943 8404 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:25.502002 8404 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:54:25.502058 8404 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:54:25.516172 8404 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=34645
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:25.518286 8404 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:25.520280 8404 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:25.535212 8410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:25.535171 8411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:25.535501 8413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:25.536679 8404 server_base.cc:1061] running on GCE node
I20260620 01:54:25.537703 8404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:25.540095 8404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:25.543720 8404 hybrid_clock.cc:648] HybridClock initialized: now 1781920465543439 us; error 171 us; skew 500 ppm
I20260620 01:54:25.544345 8404 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:25 dist-test-slave-wg14 krb5kdc[7655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920465, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:25.549820 8404 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:54:25.553229 8404 webserver.cc:533] Webserver started at http://127.20.1.190:34645/ using document root <none> and password file <none>
I20260620 01:54:25.554455 8404 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:25.554672 8404 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:25.562644 8404 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:54:25.567701 8420 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:25.570356 8404 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260620 01:54:25.570680 8404 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "f618f87e8d484910a4586badc3211eb0"
format_stamp: "Formatted at 2026-06-20 01:54:19 on dist-test-slave-wg14"
I20260620 01:54:25.571864 8404 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:25.599206 8404 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:25.613399 8404 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:25.613734 8404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:25.643736 8404 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:42071
I20260620 01:54:25.643740 8471 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:42071 every 8 connection(s)
I20260620 01:54:25.645732 8404 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:54:25.650183 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 8404
I20260620 01:54:25.650731 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 7963
I20260620 01:54:25.659351 8472 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:54:25.664499 8472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: Bootstrap starting.
Jun 20 01:54:25 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 01:54:25.678812 8100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:54:25.692853 8472 log.cc:826] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: Log is configured to *not* fsync() on all Append() calls
I20260620 01:54:25.710943 8472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:54:25.711902 8472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: Bootstrap complete.
I20260620 01:54:25.722074 8472 raft_consensus.cc:359] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:25.723263 8472 raft_consensus.cc:740] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f618f87e8d484910a4586badc3211eb0, State: Initialized, Role: FOLLOWER
I20260620 01:54:25.724372 8472 consensus_queue.cc:260] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:25.725672 8478 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:25.726292 8478 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:25.727382 8472 sys_catalog.cc:565] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:54:25.746841 8489 catalog_manager.cc:1294] Loaded cluster ID: 7c89cc2f428d4a57adaab580812f43ad
I20260620 01:54:25.747063 8489 catalog_manager.cc:1587] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: loading cluster ID for follower catalog manager: success
I20260620 01:54:25.752966 8489 catalog_manager.cc:1609] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: acquiring CA information for follower catalog manager: success
I20260620 01:54:25.756832 8489 catalog_manager.cc:1637] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:54:25.794811 8393 heartbeater.cc:499] Master 127.20.1.190:42071 was elected leader, sending a full tablet report...
I20260620 01:54:25.822889 8437 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "483ef5d33e474f5290b962ec36c02a45" instance_seqno: 1781920464712734) as {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:53509; Asking this server to re-register.
I20260620 01:54:25.827085 8393 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:25.830240 8437 ts_manager.cc:194] Registered new tserver with Master: 483ef5d33e474f5290b962ec36c02a45 (127.20.1.131:36341)
I20260620 01:54:25.835290 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 7963
I20260620 01:54:25.841311 8404 master_runner.cc:264] No leader master found from master f618f87e8d484910a4586badc3211eb0
I20260620 01:54:25.841500 8404 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:54:25.853710 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:36051
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--webserver_port=40753
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:25.933708 8497 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f618f87e8d484910a4586badc3211eb0)
I20260620 01:54:25.934038 8497 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:25.939304 7796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6edf4a403b194538857d38331fb4c16d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9322a24f7def435a92159114e88b9d7c" is_pre_election: true
I20260620 01:54:25.941032 8497 leader_election.cc:290] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f618f87e8d484910a4586badc3211eb0 (127.20.1.190:42071), 9322a24f7def435a92159114e88b9d7c (127.20.1.189:37363)
I20260620 01:54:25.952976 8447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6edf4a403b194538857d38331fb4c16d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f618f87e8d484910a4586badc3211eb0" is_pre_election: true
I20260620 01:54:25.953940 8447 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edf4a403b194538857d38331fb4c16d in term 1.
I20260620 01:54:25.955195 7854 leader_election.cc:304] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6edf4a403b194538857d38331fb4c16d, f618f87e8d484910a4586badc3211eb0; no voters: 9322a24f7def435a92159114e88b9d7c
I20260620 01:54:25.955860 8497 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:54:25.956019 8497 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 1 FOLLOWER]: Starting leader election (detected failure of leader f618f87e8d484910a4586badc3211eb0)
I20260620 01:54:25.956128 8497 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:54:25.959597 8497 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:25.960640 8497 leader_election.cc:290] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [CANDIDATE]: Term 2 election: Requested vote from peers f618f87e8d484910a4586badc3211eb0 (127.20.1.190:42071), 9322a24f7def435a92159114e88b9d7c (127.20.1.189:37363)
I20260620 01:54:25.961452 7796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6edf4a403b194538857d38331fb4c16d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9322a24f7def435a92159114e88b9d7c"
I20260620 01:54:25.961889 8447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6edf4a403b194538857d38331fb4c16d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f618f87e8d484910a4586badc3211eb0"
I20260620 01:54:25.962466 8447 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:54:25.970023 8447 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edf4a403b194538857d38331fb4c16d in term 2.
I20260620 01:54:25.971189 7854 leader_election.cc:304] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6edf4a403b194538857d38331fb4c16d, f618f87e8d484910a4586badc3211eb0; no voters: 9322a24f7def435a92159114e88b9d7c
I20260620 01:54:25.971818 8497 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 01:54:25.972887 8497 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 2 LEADER]: Becoming Leader. State: Replica: 6edf4a403b194538857d38331fb4c16d, State: Running, Role: LEADER
I20260620 01:54:25.973688 8497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:25.976918 8499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6edf4a403b194538857d38331fb4c16d. Latest consensus state: current_term: 2 leader_uuid: "6edf4a403b194538857d38331fb4c16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:25.977398 8499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: This master's current role is: LEADER
I20260620 01:54:25.979628 8501 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:54:25.982962 8501 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:54:25.984522 8501 catalog_manager.cc:1294] Loaded cluster ID: 7c89cc2f428d4a57adaab580812f43ad
I20260620 01:54:25.984699 8501 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:54:25.986655 8501 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:54:25.988168 8501 catalog_manager.cc:6080] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: Loaded TSK: 0
I20260620 01:54:25.990003 8501 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:54:26.009097 8250 heartbeater.cc:344] Connected to a master server at 127.20.1.190:42071
I20260620 01:54:26.009481 8250 heartbeater.cc:499] Master 127.20.1.190:42071 was elected leader, sending a full tablet report...
I20260620 01:54:26.011229 8437 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "67f8a95b2d2d4d3db088a4777111b11e" instance_seqno: 1781920463886509) as {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:49203; Asking this server to re-register.
I20260620 01:54:26.012343 8250 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:26.014958 8437 ts_manager.cc:194] Registered new tserver with Master: 67f8a95b2d2d4d3db088a4777111b11e (127.20.1.130:43911)
I20260620 01:54:26.101639 8503 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f618f87e8d484910a4586badc3211eb0)
I20260620 01:54:26.102187 8503 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:26.107962 7877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9322a24f7def435a92159114e88b9d7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6edf4a403b194538857d38331fb4c16d" is_pre_election: true
I20260620 01:54:26.110586 8503 leader_election.cc:290] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f618f87e8d484910a4586badc3211eb0 (127.20.1.190:42071), 6edf4a403b194538857d38331fb4c16d (127.20.1.188:46173)
I20260620 01:54:26.124099 8447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9322a24f7def435a92159114e88b9d7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f618f87e8d484910a4586badc3211eb0" is_pre_election: true
I20260620 01:54:26.124782 8447 raft_consensus.cc:2393] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9322a24f7def435a92159114e88b9d7c in current term 2: Already voted for candidate 6edf4a403b194538857d38331fb4c16d in this term.
I20260620 01:54:26.126497 7775 leader_election.cc:304] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9322a24f7def435a92159114e88b9d7c; no voters: 6edf4a403b194538857d38331fb4c16d, f618f87e8d484910a4586badc3211eb0
I20260620 01:54:26.127556 8503 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:54:26.132339 8503 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 01:54:26.402894 8496 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:26.403379 8496 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:26.403530 8496 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:26.415444 8496 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:26.415721 8496 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:54:26.429616 8496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:36051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=40753
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:26.432190 8496 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:26.434203 8496 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:26.450510 8509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:26.450469 8510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:26.451061 8512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:26.452662 8496 server_base.cc:1061] running on GCE node
I20260620 01:54:26.453671 8496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:26.455613 8496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:26.459460 8496 hybrid_clock.cc:648] HybridClock initialized: now 1781920466459024 us; error 211 us; skew 500 ppm
I20260620 01:54:26.460090 8496 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:26 dist-test-slave-wg14 krb5kdc[7655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920466, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:26.466135 8496 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:54:26.472388 8496 webserver.cc:533] Webserver started at http://127.20.1.129:40753/ using document root <none> and password file <none>
I20260620 01:54:26.473507 8496 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:26.473699 8496 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:26.483888 8496 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.003s sys 0.004s
I20260620 01:54:26.490289 8520 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:26.493219 8496 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
I20260620 01:54:26.493568 8496 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "cbec150f8e594225b4d792c3284bb950"
format_stamp: "Formatted at 2026-06-20 01:54:23 on dist-test-slave-wg14"
I20260620 01:54:26.494737 8496 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:26.495313 7796 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 2 FOLLOWER]: Refusing update from remote peer 6edf4a403b194538857d38331fb4c16d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:54:26.496567 8521 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:54:26.499702 8503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6edf4a403b194538857d38331fb4c16d. Latest consensus state: current_term: 2 leader_uuid: "6edf4a403b194538857d38331fb4c16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:26.500051 8503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:26.504129 8523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6edf4a403b194538857d38331fb4c16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:26.504660 8523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: This master's current role is: LEADER
I20260620 01:54:26.506235 8447 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 2 FOLLOWER]: Refusing update from remote peer 6edf4a403b194538857d38331fb4c16d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:54:26.510401 8521 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:54:26.513749 8503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6edf4a403b194538857d38331fb4c16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:26.514137 8503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:26.517295 8535 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6edf4a403b194538857d38331fb4c16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:26.518563 8535 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:26.518555 8530 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6edf4a403b194538857d38331fb4c16d. Latest consensus state: current_term: 2 leader_uuid: "6edf4a403b194538857d38331fb4c16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:26.518945 8530 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:26.526221 8496 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:26.531231 8496 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:26.531517 8496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:26.533154 8496 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:26.535630 8496 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:26.535796 8496 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:26.535928 8496 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:26.536010 8496 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:26.580287 8496 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:36051
I20260620 01:54:26.580324 8643 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:36051 every 8 connection(s)
I20260620 01:54:26.582401 8496 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:54:26.589147 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 8496
Jun 20 01:54:26 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920466, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:54:26 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920466, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:54:26 dist-test-slave-wg14 krb5kdc[7655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920466, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:54:26.629976 8645 heartbeater.cc:344] Connected to a master server at 127.20.1.190:42071
I20260620 01:54:26.630573 8645 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:26.632570 8645 heartbeater.cc:507] Master 127.20.1.190:42071 requested a full tablet report, sending...
I20260620 01:54:26.632789 8646 heartbeater.cc:344] Connected to a master server at 127.20.1.189:37363
I20260620 01:54:26.633044 8646 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:26.633852 8646 heartbeater.cc:507] Master 127.20.1.189:37363 requested a full tablet report, sending...
I20260620 01:54:26.630007 8644 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46173
I20260620 01:54:26.634276 8644 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:26.634878 8644 heartbeater.cc:507] Master 127.20.1.188:46173 requested a full tablet report, sending...
I20260620 01:54:26.635893 7788 ts_manager.cc:194] Re-registered known tserver with Master: cbec150f8e594225b4d792c3284bb950 (127.20.1.129:36051)
I20260620 01:54:26.636488 8437 ts_manager.cc:194] Registered new tserver with Master: cbec150f8e594225b4d792c3284bb950 (127.20.1.129:36051)
I20260620 01:54:26.636565 7867 ts_manager.cc:194] Re-registered known tserver with Master: cbec150f8e594225b4d792c3284bb950 (127.20.1.129:36051)
I20260620 01:54:26.638571 7867 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:41271
I20260620 01:54:26.658739 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 8496
W20260620 01:54:26.660009 8656 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:26.655557 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:36138 (local address 127.20.1.129:36051)
0620 01:54:26.656243 (+ 686us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:26.656251 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:26.656330 (+ 79us) server_negotiation.cc:408] Connection header received
0620 01:54:26.656412 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:26.656421 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:26.656712 (+ 291us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:26.656932 (+ 220us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:26.658216 (+ 1284us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36138: BlockingRecv error: recv got EOF from 127.0.0.1:36138 (error 108)
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":345,"threads_started":1}
W20260620 01:54:26.664904 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 8496: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:36051: server requires authentication, but client does not have Kerberos credentials available
W20260620 01:54:26.665099 8656 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:26.661452 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:36142 (local address 127.20.1.129:36051)
0620 01:54:26.661770 (+ 318us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:26.661777 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:26.661958 (+ 181us) server_negotiation.cc:408] Connection header received
0620 01:54:26.662491 (+ 533us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:26.662497 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:26.662587 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:26.662917 (+ 330us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:26.664646 (+ 1729us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36142: BlockingRecv error: recv got EOF from 127.0.0.1:36142 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
I20260620 01:54:26.665169 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 8496
I20260620 01:54:26.685032 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:36051
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--webserver_port=40753
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:26.949867 8521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6edf4a403b194538857d38331fb4c16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:26.950443 8521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: This master's current role is: LEADER
I20260620 01:54:27.019924 8249 heartbeater.cc:499] Master 127.20.1.188:46173 was elected leader, sending a full tablet report...
W20260620 01:54:27.177242 8657 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:27.177778 8657 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:27.177976 8657 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:54:27.178071 8657 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:27.190996 8657 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:27.191423 8657 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:54:27.212127 8657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:36051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=40753
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:27.214570 8657 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:27.216578 8657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:27.230558 8663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:27.230545 8664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:27.231560 8666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:27.231858 8657 server_base.cc:1061] running on GCE node
I20260620 01:54:27.233309 8657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:27.235311 8657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:27.239202 8657 hybrid_clock.cc:648] HybridClock initialized: now 1781920467238844 us; error 157 us; skew 500 ppm
I20260620 01:54:27.239880 8657 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:27 dist-test-slave-wg14 krb5kdc[7655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920467, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:27.244865 8657 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:54:27.247823 8657 webserver.cc:533] Webserver started at http://127.20.1.129:40753/ using document root <none> and password file <none>
I20260620 01:54:27.248915 8657 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:27.249356 8657 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:27.258038 8657 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:54:27.264173 8673 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:27.267908 8657 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.002s sys 0.002s
I20260620 01:54:27.268270 8657 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "cbec150f8e594225b4d792c3284bb950"
format_stamp: "Formatted at 2026-06-20 01:54:23 on dist-test-slave-wg14"
I20260620 01:54:27.269411 8657 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:27.295506 8657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:27.302574 8657 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:27.302932 8657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:27.304270 8657 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:27.307004 8657 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:27.307206 8657 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 01:54:27.307399 8657 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:27.307471 8657 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:54:27.307976 8657 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:54:27.308050 8657 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:54:27.308140 8657 fault_injection.cc:59] injecting fault for kudu (pid 8657): FLAGS_fault_before_start (process will exit)
I20260620 01:54:27.332986 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:36051
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--webserver_port=40753
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:27.795713 8392 heartbeater.cc:499] Master 127.20.1.188:46173 was elected leader, sending a full tablet report...
W20260620 01:54:27.804592 8681 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:27.805173 8681 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:27.805306 8681 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:27.816687 8681 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:27.816987 8681 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:54:27.830839 8681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33669,127.20.1.149:42889,127.20.1.150:46823,127.20.1.151:46257,127.20.1.152:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:36051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=40753
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:27.833346 8681 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:27.835479 8681 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:27.850903 8687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:27.850890 8686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:27.854779 8681 server_base.cc:1061] running on GCE node
W20260620 01:54:27.855825 8689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:27.856877 8681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:27.858551 8681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:27.862109 8681 hybrid_clock.cc:648] HybridClock initialized: now 1781920467861805 us; error 207 us; skew 500 ppm
I20260620 01:54:27.862658 8681 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:27 dist-test-slave-wg14 krb5kdc[7655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920467, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:27.867358 8681 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:54:27.870054 8681 webserver.cc:533] Webserver started at http://127.20.1.129:40753/ using document root <none> and password file <none>
I20260620 01:54:27.871203 8681 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:27.871393 8681 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:27.879081 8681 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:54:27.884198 8696 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:27.887012 8681 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.000s sys 0.004s
I20260620 01:54:27.887347 8681 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "cbec150f8e594225b4d792c3284bb950"
format_stamp: "Formatted at 2026-06-20 01:54:23 on dist-test-slave-wg14"
I20260620 01:54:27.888391 8681 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:27.914383 8681 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:27.920135 8681 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:27.920490 8681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:27.922454 8681 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:27.925048 8681 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:27.925230 8681 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:27.925367 8681 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:27.925450 8681 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:27.971664 8681 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:36051
I20260620 01:54:27.971925 8808 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:36051 every 8 connection(s)
I20260620 01:54:27.973856 8681 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:54:27.980252 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 8681
I20260620 01:54:27.980685 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 8681
W20260620 01:54:28.004904 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 8681: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:36051: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:54:28.005122 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 8681
I20260620 01:54:28.023365 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 8119
W20260620 01:54:28.024333 8818 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:27.993521 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:42013 (local address 127.20.1.188:46173)
0620 01:54:27.998443 (+ 4922us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:27.998451 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:28.001366 (+ 2915us) server_negotiation.cc:408] Connection header received
0620 01:54:28.001486 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:28.001492 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:28.001604 (+ 112us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:28.001784 (+ 180us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:28.023040 (+ 21256us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:42013: BlockingRecv error: recv got EOF from 127.20.1.129:42013 (error 108)
Metrics: {"server-negotiator.queue_time_us":4770,"thread_start_us":5330,"threads_started":1}
W20260620 01:54:28.025259 8814 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:27.978571 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:53345 (local address 127.20.1.190:42071)
0620 01:54:27.999995 (+ 21424us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:28.000005 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:28.000044 (+ 39us) server_negotiation.cc:408] Connection header received
0620 01:54:28.000197 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:28.000206 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:28.000339 (+ 133us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:28.002657 (+ 2318us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:28.023094 (+ 20437us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:53345: BlockingRecv error: recv got EOF from 127.20.1.129:53345 (error 108)
Metrics: {"server-negotiator.queue_time_us":21215,"thread_start_us":21057,"threads_started":1}
W20260620 01:54:28.027282 8813 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:27.978400 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:33147 (local address 127.20.1.189:37363)
0620 01:54:28.003529 (+ 25129us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:28.003537 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:28.003562 (+ 25us) server_negotiation.cc:408] Connection header received
0620 01:54:28.003649 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:28.003656 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:28.003758 (+ 102us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:28.004446 (+ 688us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:28.025710 (+ 21264us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:33147: BlockingRecv error: recv got EOF from 127.20.1.129:33147 (error 108)
Metrics: {"server-negotiator.queue_time_us":24961,"thread_start_us":24773,"threads_started":1}
W20260620 01:54:28.028362 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 8119: Not authorized: Client connection negotiation failed: client connection to 127.20.1.130:43911: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:54:28.028685 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 8119
I20260620 01:54:28.046190 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 8263
W20260620 01:54:28.050447 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 8263: Not authorized: Client connection negotiation failed: client connection to 127.20.1.131:36341: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:54:28.050650 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 8263
I20260620 01:54:28.066283 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 8404
W20260620 01:54:28.069723 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 8404: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:42071: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:54:28.069912 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 8404
W20260620 01:54:28.069892 8814 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:28.067750 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.20.1.190:42071)
0620 01:54:28.068070 (+ 320us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:28.068076 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:28.068106 (+ 30us) server_negotiation.cc:408] Connection header received
0620 01:54:28.068210 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:28.068216 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:28.068314 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:28.068473 (+ 159us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:28.069615 (+ 1142us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60332: BlockingRecv error: recv got EOF from 127.0.0.1:60332 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
I20260620 01:54:28.092017 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 7756
I20260620 01:54:28.094023 7818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:54:28.236809 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 7756
I20260620 01:54:28.259917 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 7833
I20260620 01:54:28.262408 7898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:54:28.405753 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 7833
2026-06-20T01:54:28Z chronyd exiting
2026-06-20T01:54:28Z chronyd exiting
2026-06-20T01:54:28Z chronyd exiting
2026-06-20T01:54:28Z chronyd exiting
2026-06-20T01:54:28Z chronyd exiting
Jun 20 01:54:28 dist-test-slave-wg14 krb5kdc[8830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 01:54:28 dist-test-slave-wg14 krb5kdc[8830](info): set up 2 sockets
Jun 20 01:54:28 dist-test-slave-wg14 krb5kdc[8830](info): commencing operation
krb5kdc: starting...
W20260620 01:54:30.508664 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.022s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 01:54:30 dist-test-slave-wg14 krb5kdc[8830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T01:54:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:54:30Z Disabled control of system clock
2026-06-20T01:54:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:54:30Z Disabled control of system clock
2026-06-20T01:54:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:54:30Z Disabled control of system clock
2026-06-20T01:54:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:54:30Z Disabled control of system clock
2026-06-20T01:54:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:54:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.190@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.190@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:31.052228 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42071
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:54:31.645701 8854 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:31.646127 8854 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:31.646236 8854 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:31.658783 8854 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:54:31.658951 8854 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:31.659013 8854 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:54:31.659055 8854 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:54:31.674887 8854 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:31.676944 8854 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:31.678675 8854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:31.697554 8860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:31.697443 8859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:31.698262 8862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:31.698971 8854 server_base.cc:1061] running on GCE node
I20260620 01:54:31.700783 8854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:31.703522 8854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:31.707363 8854 hybrid_clock.cc:648] HybridClock initialized: now 1781920471707034 us; error 238 us; skew 500 ppm
I20260620 01:54:31.708271 8854 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:31 dist-test-slave-wg14 krb5kdc[8830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920471, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:31.713919 8854 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:54:31.716593 8854 webserver.cc:533] Webserver started at http://127.20.1.190:44455/ using document root <none> and password file <none>
I20260620 01:54:31.717572 8854 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:31.717756 8854 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:31.731940 8854 fs_manager.cc:714] Time spent opening directory manager: real 0.012s user 0.008s sys 0.004s
I20260620 01:54:31.736763 8869 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:31.739022 8854 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:54:31.739374 8854 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "f618f87e8d484910a4586badc3211eb0"
format_stamp: "Formatted at 2026-06-20 01:54:19 on dist-test-slave-wg14"
I20260620 01:54:31.740398 8854 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:31.769414 8854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:31.780295 8854 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:31.781224 8854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:31.817464 8854 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:42071
I20260620 01:54:31.817345 8921 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:42071 every 8 connection(s)
I20260620 01:54:31.819952 8854 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:54:31.823601 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 8854
I20260620 01:54:31.823863 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:54:31.834206 8922 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:54:31.839568 8922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: Bootstrap starting.
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.189@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:54:31.869499 8922 log.cc:826] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: Log is configured to *not* fsync() on all Append() calls
I20260620 01:54:31.885980 8922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:54:31.886853 8922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: Bootstrap complete.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:31.899463 8922 raft_consensus.cc:359] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:31.900918 8922 raft_consensus.cc:740] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f618f87e8d484910a4586badc3211eb0, State: Initialized, Role: FOLLOWER
I20260620 01:54:31.902156 8922 consensus_queue.cc:260] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:31.903623 8927 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:31.904179 8927 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:31.905700 8922 sys_catalog.cc:565] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:54:31.926071 8939 catalog_manager.cc:1294] Loaded cluster ID: 7c89cc2f428d4a57adaab580812f43ad
I20260620 01:54:31.926405 8939 catalog_manager.cc:1587] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: loading cluster ID for follower catalog manager: success
I20260620 01:54:31.933687 8939 catalog_manager.cc:1609] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: acquiring CA information for follower catalog manager: success
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.189@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:54:31.940008 8939 catalog_manager.cc:1637] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:54:31.958225 8854 master_runner.cc:187] Error getting master registration for 127.20.1.189:37363: Network error: Client connection negotiation failed: client connection to 127.20.1.189:37363: connect: Connection refused (error 111),
I20260620 01:54:31.970199 8854 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:31.994980 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:37363
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:54:32.512856 8944 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:32.513638 8944 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:32.513880 8944 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:32.525348 8944 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:54:32.525503 8944 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:32.525563 8944 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:54:32.525604 8944 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:54:32.542497 8944 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:37363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:32.544653 8944 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:32.546492 8944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:32.558559 8950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:32.559538 8949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:32.561259 8952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:32.562282 8944 server_base.cc:1061] running on GCE node
I20260620 01:54:32.563841 8944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:32.566325 8944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:32.569902 8944 hybrid_clock.cc:648] HybridClock initialized: now 1781920472569626 us; error 183 us; skew 500 ppm
I20260620 01:54:32.570554 8944 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:32 dist-test-slave-wg14 krb5kdc[8830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920472, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:32.577816 8944 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 01:54:32.581802 8944 webserver.cc:533] Webserver started at http://127.20.1.189:36581/ using document root <none> and password file <none>
I20260620 01:54:32.583034 8944 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:32.583232 8944 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:32.591267 8944 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 01:54:32.597425 8959 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:32.600622 8944 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:54:32.600925 8944 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "9322a24f7def435a92159114e88b9d7c"
format_stamp: "Formatted at 2026-06-20 01:54:20 on dist-test-slave-wg14"
I20260620 01:54:32.601960 8944 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:32.621917 8944 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:32.628882 8944 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:32.629221 8944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:32.665433 8944 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:37363
I20260620 01:54:32.665475 9011 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:37363 every 8 connection(s)
I20260620 01:54:32.667826 8944 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:54:32.675252 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 8944
I20260620 01:54:32.675527 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:54:32.684039 9012 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:54:32.694108 9012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c: Bootstrap starting.
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.188@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:54:32.730999 9012 log.cc:826] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c: Log is configured to *not* fsync() on all Append() calls
I20260620 01:54:32.746258 9012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:54:32.746963 9012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c: Bootstrap complete.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:32.758468 9012 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:32.759750 9012 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9322a24f7def435a92159114e88b9d7c, State: Initialized, Role: FOLLOWER
I20260620 01:54:32.760867 9012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:32.762468 9017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:32.763000 9017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:32.765437 9012 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: configured and running, proceeding with master startup.
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.188@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:54:32.787698 9029 catalog_manager.cc:1294] Loaded cluster ID: 7c89cc2f428d4a57adaab580812f43ad
I20260620 01:54:32.788009 9029 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c: loading cluster ID for follower catalog manager: success
I20260620 01:54:32.794158 9029 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c: acquiring CA information for follower catalog manager: success
I20260620 01:54:32.798580 9029 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Jun 20 01:54:32 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920472, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:32.839685 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:46173
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:32.849711 8944 master_runner.cc:187] Error getting master registration for 127.20.1.188:46173: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46173: connect: Connection refused (error 111),
I20260620 01:54:32.856940 8944 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:54:33.080430 9036 raft_consensus.cc:493] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:54:33.080937 9036 raft_consensus.cc:515] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:33.085232 9036 leader_election.cc:290] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9322a24f7def435a92159114e88b9d7c (127.20.1.189:37363), 6edf4a403b194538857d38331fb4c16d (127.20.1.188:46173)
W20260620 01:54:33.102138 8872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46173: connect: Connection refused (error 111)
I20260620 01:54:33.105123 8987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f618f87e8d484910a4586badc3211eb0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9322a24f7def435a92159114e88b9d7c" is_pre_election: true
W20260620 01:54:33.105448 8872 leader_election.cc:336] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6edf4a403b194538857d38331fb4c16d (127.20.1.188:46173): Network error: Client connection negotiation failed: client connection to 127.20.1.188:46173: connect: Connection refused (error 111)
I20260620 01:54:33.105916 8987 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f618f87e8d484910a4586badc3211eb0 in term 2.
I20260620 01:54:33.106899 8872 leader_election.cc:304] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9322a24f7def435a92159114e88b9d7c, f618f87e8d484910a4586badc3211eb0; no voters: 6edf4a403b194538857d38331fb4c16d
I20260620 01:54:33.107798 9036 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 01:54:33.108057 9036 raft_consensus.cc:493] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:54:33.108196 9036 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:54:33.117496 9036 raft_consensus.cc:515] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:33.118791 9036 leader_election.cc:290] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [CANDIDATE]: Term 3 election: Requested vote from peers 9322a24f7def435a92159114e88b9d7c (127.20.1.189:37363), 6edf4a403b194538857d38331fb4c16d (127.20.1.188:46173)
I20260620 01:54:33.119745 8987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f618f87e8d484910a4586badc3211eb0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9322a24f7def435a92159114e88b9d7c"
I20260620 01:54:33.120199 8987 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 2 FOLLOWER]: Advancing to term 3
W20260620 01:54:33.122180 8872 leader_election.cc:336] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6edf4a403b194538857d38331fb4c16d (127.20.1.188:46173): Network error: Client connection negotiation failed: client connection to 127.20.1.188:46173: connect: Connection refused (error 111)
I20260620 01:54:33.126943 8987 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f618f87e8d484910a4586badc3211eb0 in term 3.
I20260620 01:54:33.127913 8872 leader_election.cc:304] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9322a24f7def435a92159114e88b9d7c, f618f87e8d484910a4586badc3211eb0; no voters: 6edf4a403b194538857d38331fb4c16d
I20260620 01:54:33.128412 9036 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 01:54:33.129007 9036 raft_consensus.cc:697] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 3 LEADER]: Becoming Leader. State: Replica: f618f87e8d484910a4586badc3211eb0, State: Running, Role: LEADER
I20260620 01:54:33.130290 9036 consensus_queue.cc:237] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:33.137068 9040 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f618f87e8d484910a4586badc3211eb0. Latest consensus state: current_term: 3 leader_uuid: "f618f87e8d484910a4586badc3211eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:33.137856 9040 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: This master's current role is: LEADER
I20260620 01:54:33.139959 9042 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:54:33.143636 9042 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:54:33.145051 9042 catalog_manager.cc:1294] Loaded cluster ID: 7c89cc2f428d4a57adaab580812f43ad
I20260620 01:54:33.145231 9042 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:54:33.147047 9042 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:54:33.148629 9042 catalog_manager.cc:6080] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: Loaded TSK: 0
I20260620 01:54:33.150436 9042 catalog_manager.cc:1549] Initializing in-progress tserver states...
W20260620 01:54:33.373018 9035 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:33.373518 9035 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:33.373654 9035 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:33.386710 9035 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:54:33.386857 9035 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:33.386915 9035 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:54:33.386956 9035 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:54:33.403887 9035 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:46173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:33.406333 9035 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:33.408320 9035 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:33.421753 9047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:33.422510 9048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:33.422665 9050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:33.425011 9035 server_base.cc:1061] running on GCE node
I20260620 01:54:33.426213 9035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:33.428804 9035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:33.432473 9035 hybrid_clock.cc:648] HybridClock initialized: now 1781920473432221 us; error 165 us; skew 500 ppm
I20260620 01:54:33.433300 9035 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:33 dist-test-slave-wg14 krb5kdc[8830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920473, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:33.440219 9035 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 01:54:33.442842 9035 webserver.cc:533] Webserver started at http://127.20.1.188:38491/ using document root <none> and password file <none>
I20260620 01:54:33.443933 9035 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:33.444110 9035 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:33.453646 9035 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 01:54:33.459960 9057 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:33.462519 9035 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s
I20260620 01:54:33.462785 9035 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "6edf4a403b194538857d38331fb4c16d"
format_stamp: "Formatted at 2026-06-20 01:54:21 on dist-test-slave-wg14"
I20260620 01:54:33.463858 9035 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:33.493163 9035 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:33.500526 9035 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:33.500957 9035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:33.532979 9035 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:46173
I20260620 01:54:33.532991 9108 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:46173 every 8 connection(s)
I20260620 01:54:33.535300 9035 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:54:33.535806 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 9035
I20260620 01:54:33.536021 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:54:33.547545 9109 sys_catalog.cc:263] Verifying existing consensus state
Jun 20 01:54:33 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:54:33.552769 9109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: Bootstrap starting.
I20260620 01:54:33.579042 9109 log.cc:826] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: Log is configured to *not* fsync() on all Append() calls
Jun 20 01:54:33 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:54:33 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920471, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:54:33.598767 9109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:54:33.600144 9109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: Bootstrap complete.
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.129@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:54:33.612046 9109 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:33.613654 9109 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edf4a403b194538857d38331fb4c16d, State: Initialized, Role: FOLLOWER
I20260620 01:54:33.615051 9109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:33.617292 9117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:33.617853 9117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:33.619292 9109 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:54:33.619894 9084 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:54:33.634748 9084 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 3 FOLLOWER]: Refusing update from remote peer f618f87e8d484910a4586badc3211eb0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:54:33.638026 9040 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
Jun 20 01:54:33 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:54:33.648551 9117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f618f87e8d484910a4586badc3211eb0. Latest consensus state: current_term: 3 leader_uuid: "f618f87e8d484910a4586badc3211eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:33.650511 9117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:33.684649 8987 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 3 FOLLOWER]: Refusing update from remote peer f618f87e8d484910a4586badc3211eb0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:54:33.684938 9131 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f618f87e8d484910a4586badc3211eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:33.685351 9131 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: This master's current role is: LEADER
I20260620 01:54:33.686676 9133 catalog_manager.cc:1294] Loaded cluster ID: 7c89cc2f428d4a57adaab580812f43ad
I20260620 01:54:33.686825 9133 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: loading cluster ID for follower catalog manager: success
I20260620 01:54:33.689917 9132 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:54:33.691411 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.124s user 0.002s sys 0.029s
I20260620 01:54:33.712980 9133 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: acquiring CA information for follower catalog manager: success
I20260620 01:54:33.714928 9117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f618f87e8d484910a4586badc3211eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:33.715288 9117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 01:54:33 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920473, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:54:33.720248 9133 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:54:33.720793 9143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f618f87e8d484910a4586badc3211eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:33.721886 9143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:33.722334 9139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f618f87e8d484910a4586badc3211eb0. Latest consensus state: current_term: 3 leader_uuid: "f618f87e8d484910a4586badc3211eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:33.722648 9139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: This master's current role is: FOLLOWER
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.129@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:33.784435 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:34.249657 9131 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f618f87e8d484910a4586badc3211eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:34.250243 9131 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: This master's current role is: LEADER
W20260620 01:54:34.292804 9145 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:34.293282 9145 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:34.293449 9145 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:34.306867 9145 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:34.307190 9145 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:54:34.323879 9145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:34.326028 9145 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:34.327919 9145 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:34.345256 9153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:34.345256 9154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:34.346273 9156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:34.346401 9145 server_base.cc:1061] running on GCE node
I20260620 01:54:34.347332 9145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:34.349004 9145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:34.352685 9145 hybrid_clock.cc:648] HybridClock initialized: now 1781920474352372 us; error 204 us; skew 500 ppm
I20260620 01:54:34.353223 9145 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:34 dist-test-slave-wg14 krb5kdc[8830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920474, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:34.360049 9145 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:54:34.363359 9145 webserver.cc:533] Webserver started at http://127.20.1.129:37433/ using document root <none> and password file <none>
I20260620 01:54:34.364445 9145 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:34.364622 9145 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:34.372963 9145 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:54:34.377674 9163 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:34.380295 9145 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:54:34.380574 9145 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "cbec150f8e594225b4d792c3284bb950"
format_stamp: "Formatted at 2026-06-20 01:54:23 on dist-test-slave-wg14"
I20260620 01:54:34.381904 9145 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:34.414881 9145 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:34.420428 9145 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:34.420893 9145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:34.422230 9145 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:34.424692 9145 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:34.424857 9145 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:34.424988 9145 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:34.425071 9145 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:34.481818 9276 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:39993 every 8 connection(s)
I20260620 01:54:34.481766 9145 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:39993
I20260620 01:54:34.484105 9145 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:54:34.489866 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 9145
I20260620 01:54:34.490487 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 01:54:34 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920474, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:54:34 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920474, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.130@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.130@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:54:34.554806 9279 heartbeater.cc:344] Connected to a master server at 127.20.1.189:37363
I20260620 01:54:34.555222 9278 heartbeater.cc:344] Connected to a master server at 127.20.1.190:42071
I20260620 01:54:34.555482 9278 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:34.555485 9279 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:34.557335 9279 heartbeater.cc:507] Master 127.20.1.189:37363 requested a full tablet report, sending...
Jun 20 01:54:34 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920474, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:54:34.561461 8976 ts_manager.cc:194] Registered new tserver with Master: cbec150f8e594225b4d792c3284bb950 (127.20.1.129:39993)
I20260620 01:54:34.563813 9278 heartbeater.cc:507] Master 127.20.1.190:42071 requested a full tablet report, sending...
I20260620 01:54:34.567022 8886 ts_manager.cc:194] Registered new tserver with Master: cbec150f8e594225b4d792c3284bb950 (127.20.1.129:39993)
I20260620 01:54:34.569723 8886 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:49361
I20260620 01:54:34.574018 9277 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46173
I20260620 01:54:34.574249 9277 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:34.574699 9277 heartbeater.cc:507] Master 127.20.1.188:46173 requested a full tablet report, sending...
I20260620 01:54:34.577345 9074 ts_manager.cc:194] Registered new tserver with Master: cbec150f8e594225b4d792c3284bb950 (127.20.1.129:39993)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.130 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.130 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.130@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.130@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.130 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.130 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:34.656601 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:54:35.131964 9291 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:35.132429 9291 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:35.132586 9291 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:35.143086 9291 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:35.143407 9291 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:54:35.157199 9291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:35.159350 9291 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:35.161031 9291 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:35.174695 9296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:35.174783 9299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:35.174661 9297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:35.176471 9291 server_base.cc:1061] running on GCE node
I20260620 01:54:35.177443 9291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:35.179270 9291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:35.182961 9291 hybrid_clock.cc:648] HybridClock initialized: now 1781920475182690 us; error 161 us; skew 500 ppm
I20260620 01:54:35.183665 9291 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:35 dist-test-slave-wg14 krb5kdc[8830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920475, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:35.190285 9291 init.cc:377] Logged in from keytab as kudu/127.20.1.130@KRBTEST.COM (short username kudu)
I20260620 01:54:35.193492 9291 webserver.cc:533] Webserver started at http://127.20.1.130:43321/ using document root <none> and password file <none>
I20260620 01:54:35.194808 9291 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:35.195461 9291 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:35.205417 9291 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 01:54:35.211004 9306 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:35.214075 9291 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.001s sys 0.002s
I20260620 01:54:35.214354 9291 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "67f8a95b2d2d4d3db088a4777111b11e"
format_stamp: "Formatted at 2026-06-20 01:54:23 on dist-test-slave-wg14"
I20260620 01:54:35.216131 9291 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:35.256604 9291 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:35.263687 9291 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:35.264045 9291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:35.265491 9291 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:35.268100 9291 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:35.268249 9291 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:35.268373 9291 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:35.268457 9291 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:35.320130 9291 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:35511
I20260620 01:54:35.320209 9419 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:35511 every 8 connection(s)
I20260620 01:54:35.322620 9291 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:54:35.323478 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 9291
I20260620 01:54:35.323804 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
Jun 20 01:54:35 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920475, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:54:35 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920475, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.131@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.131@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Jun 20 01:54:35 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920475, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:54:35.386957 9422 heartbeater.cc:344] Connected to a master server at 127.20.1.189:37363
I20260620 01:54:35.387631 9422 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:35.389024 9422 heartbeater.cc:507] Master 127.20.1.189:37363 requested a full tablet report, sending...
I20260620 01:54:35.392093 8976 ts_manager.cc:194] Registered new tserver with Master: 67f8a95b2d2d4d3db088a4777111b11e (127.20.1.130:35511)
I20260620 01:54:35.392282 9421 heartbeater.cc:344] Connected to a master server at 127.20.1.190:42071
I20260620 01:54:35.392470 9421 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:35.393467 9421 heartbeater.cc:507] Master 127.20.1.190:42071 requested a full tablet report, sending...
I20260620 01:54:35.395058 8886 ts_manager.cc:194] Registered new tserver with Master: 67f8a95b2d2d4d3db088a4777111b11e (127.20.1.130:35511)
I20260620 01:54:35.396488 8886 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:35351
I20260620 01:54:35.397729 9420 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46173
I20260620 01:54:35.397955 9420 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:35.399501 9420 heartbeater.cc:507] Master 127.20.1.188:46173 requested a full tablet report, sending...
I20260620 01:54:35.403249 9074 ts_manager.cc:194] Registered new tserver with Master: 67f8a95b2d2d4d3db088a4777111b11e (127.20.1.130:35511)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.131 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.131 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.131@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.131@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.131 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.131 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:54:35.488941 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:35.574056 9278 heartbeater.cc:499] Master 127.20.1.190:42071 was elected leader, sending a full tablet report...
W20260620 01:54:35.959434 9435 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:35.959997 9435 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:35.960170 9435 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:35.970863 9435 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:35.971187 9435 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:54:35.988462 9435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:35.990944 9435 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:35.993943 9435 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:36.011626 9441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:36.011811 9440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:36.013793 9435 server_base.cc:1061] running on GCE node
W20260620 01:54:36.014107 9443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:36.015717 9435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:36.017419 9435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:36.021052 9435 hybrid_clock.cc:648] HybridClock initialized: now 1781920476020784 us; error 118 us; skew 500 ppm
I20260620 01:54:36.021693 9435 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:36 dist-test-slave-wg14 krb5kdc[8830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920476, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:36.027627 9435 init.cc:377] Logged in from keytab as kudu/127.20.1.131@KRBTEST.COM (short username kudu)
I20260620 01:54:36.031200 9435 webserver.cc:533] Webserver started at http://127.20.1.131:35645/ using document root <none> and password file <none>
I20260620 01:54:36.032904 9435 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:36.033263 9435 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:36.047576 9435 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.006s sys 0.000s
I20260620 01:54:36.051976 9450 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:36.054638 9435 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:54:36.055055 9435 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "483ef5d33e474f5290b962ec36c02a45"
format_stamp: "Formatted at 2026-06-20 01:54:24 on dist-test-slave-wg14"
I20260620 01:54:36.056095 9435 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:36.082969 9435 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:36.089494 9435 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:36.089859 9435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:36.091184 9435 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:36.094115 9435 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:36.094291 9435 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:36.094420 9435 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:36.094498 9435 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:36.154884 9435 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:41279
I20260620 01:54:36.154959 9563 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:41279 every 8 connection(s)
I20260620 01:54:36.156863 9435 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:54:36.157406 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 9435
I20260620 01:54:36.157647 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
Jun 20 01:54:36 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920476, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:54:36 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920476, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:54:36 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920476, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:54:36.216116 9565 heartbeater.cc:344] Connected to a master server at 127.20.1.190:42071
I20260620 01:54:36.217105 9564 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46173
I20260620 01:54:36.217106 9565 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:36.217341 9564 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:36.217350 9566 heartbeater.cc:344] Connected to a master server at 127.20.1.189:37363
I20260620 01:54:36.217566 9566 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:36.218766 9566 heartbeater.cc:507] Master 127.20.1.189:37363 requested a full tablet report, sending...
I20260620 01:54:36.218811 9565 heartbeater.cc:507] Master 127.20.1.190:42071 requested a full tablet report, sending...
I20260620 01:54:36.218840 9564 heartbeater.cc:507] Master 127.20.1.188:46173 requested a full tablet report, sending...
I20260620 01:54:36.221621 8976 ts_manager.cc:194] Registered new tserver with Master: 483ef5d33e474f5290b962ec36c02a45 (127.20.1.131:41279)
I20260620 01:54:36.221691 9074 ts_manager.cc:194] Registered new tserver with Master: 483ef5d33e474f5290b962ec36c02a45 (127.20.1.131:41279)
I20260620 01:54:36.223033 8886 ts_manager.cc:194] Registered new tserver with Master: 483ef5d33e474f5290b962ec36c02a45 (127.20.1.131:41279)
I20260620 01:54:36.224166 8886 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:46965
I20260620 01:54:36.239105 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:54:36.253567 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:54:36.253837 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 8854
I20260620 01:54:36.255661 8917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:54:36.390908 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 8854
I20260620 01:54:36.401183 9421 heartbeater.cc:499] Master 127.20.1.190:42071 was elected leader, sending a full tablet report...
W20260620 01:54:36.418684 9310 connection.cc:573] client connection to 127.20.1.190:42071 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 01:54:36.419732 9421 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:42071 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:54:36.420610 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42071
--webserver_interface=127.20.1.190
--webserver_port=44455
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:54:36.579949 9278 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:42071 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:42071: connect: Connection refused (error 111)
W20260620 01:54:36.906967 9575 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:36.907397 9575 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:36.907555 9575 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:36.918638 9575 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:54:36.918797 9575 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:36.918867 9575 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:54:36.918915 9575 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:54:36.935006 9575 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=44455
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:36.937548 9575 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:36.939719 9575 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:36.953816 9581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:36.953855 9582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:36.954022 9584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:36.955027 9575 server_base.cc:1061] running on GCE node
I20260620 01:54:36.956403 9575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:36.958866 9575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:36.962422 9575 hybrid_clock.cc:648] HybridClock initialized: now 1781920476962190 us; error 132 us; skew 500 ppm
I20260620 01:54:36.963284 9575 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:36 dist-test-slave-wg14 krb5kdc[8830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920476, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:36.968348 9575 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:54:36.972035 9575 webserver.cc:533] Webserver started at http://127.20.1.190:44455/ using document root <none> and password file <none>
I20260620 01:54:36.973114 9575 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:36.973289 9575 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:36.982417 9575 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 01:54:36.987490 9591 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:36.990211 9575 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.001s sys 0.002s
I20260620 01:54:36.990543 9575 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "f618f87e8d484910a4586badc3211eb0"
format_stamp: "Formatted at 2026-06-20 01:54:19 on dist-test-slave-wg14"
I20260620 01:54:36.991779 9575 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:37.016984 9575 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:37.030555 9575 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:37.031023 9575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:37.060771 9575 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:42071
I20260620 01:54:37.060922 9643 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:42071 every 8 connection(s)
I20260620 01:54:37.063571 9575 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:54:37.065821 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 9575
I20260620 01:54:37.066252 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 9145
I20260620 01:54:37.076963 9644 sys_catalog.cc:263] Verifying existing consensus state
Jun 20 01:54:37 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 01:54:37.083465 9644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: Bootstrap starting.
I20260620 01:54:37.096899 9272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:54:37.109587 9644 log.cc:826] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: Log is configured to *not* fsync() on all Append() calls
I20260620 01:54:37.126816 9644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:54:37.127700 9644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: Bootstrap complete.
I20260620 01:54:37.136078 9644 raft_consensus.cc:359] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:37.136950 9644 raft_consensus.cc:740] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: f618f87e8d484910a4586badc3211eb0, State: Initialized, Role: FOLLOWER
I20260620 01:54:37.137995 9644 consensus_queue.cc:260] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:37.139293 9650 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:37.139791 9650 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:37.140856 9644 sys_catalog.cc:565] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:54:37.170601 9661 catalog_manager.cc:1294] Loaded cluster ID: 7c89cc2f428d4a57adaab580812f43ad
I20260620 01:54:37.170725 9661 catalog_manager.cc:1587] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: loading cluster ID for follower catalog manager: success
I20260620 01:54:37.177481 9661 catalog_manager.cc:1609] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: acquiring CA information for follower catalog manager: success
I20260620 01:54:37.182313 9661 catalog_manager.cc:1637] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:54:37.228396 9565 heartbeater.cc:499] Master 127.20.1.190:42071 was elected leader, sending a full tablet report...
I20260620 01:54:37.245203 9575 master_runner.cc:264] No leader master found from master f618f87e8d484910a4586badc3211eb0
I20260620 01:54:37.245517 9575 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:54:37.247639 9609 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "483ef5d33e474f5290b962ec36c02a45" instance_seqno: 1781920476144040) as {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:44665; Asking this server to re-register.
I20260620 01:54:37.251374 9565 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:37.254009 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 9145
I20260620 01:54:37.255927 9609 ts_manager.cc:194] Registered new tserver with Master: 483ef5d33e474f5290b962ec36c02a45 (127.20.1.131:41279)
I20260620 01:54:37.273243 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:39993
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--webserver_port=37433
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:37.437300 9421 heartbeater.cc:344] Connected to a master server at 127.20.1.190:42071
I20260620 01:54:37.438032 9421 heartbeater.cc:499] Master 127.20.1.190:42071 was elected leader, sending a full tablet report...
I20260620 01:54:37.440570 9609 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "67f8a95b2d2d4d3db088a4777111b11e" instance_seqno: 1781920475306237) as {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:59071; Asking this server to re-register.
I20260620 01:54:37.442034 9421 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:37.444460 9609 ts_manager.cc:194] Registered new tserver with Master: 67f8a95b2d2d4d3db088a4777111b11e (127.20.1.130:35511)
I20260620 01:54:37.479185 9670 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f618f87e8d484910a4586badc3211eb0)
I20260620 01:54:37.479578 9670 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:37.485176 8987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6edf4a403b194538857d38331fb4c16d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9322a24f7def435a92159114e88b9d7c" is_pre_election: true
I20260620 01:54:37.491449 9670 leader_election.cc:290] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f618f87e8d484910a4586badc3211eb0 (127.20.1.190:42071), 9322a24f7def435a92159114e88b9d7c (127.20.1.189:37363)
I20260620 01:54:37.503602 9619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6edf4a403b194538857d38331fb4c16d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f618f87e8d484910a4586badc3211eb0" is_pre_election: true
I20260620 01:54:37.504846 9619 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edf4a403b194538857d38331fb4c16d in term 3.
I20260620 01:54:37.505918 9061 leader_election.cc:304] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6edf4a403b194538857d38331fb4c16d, f618f87e8d484910a4586badc3211eb0; no voters: 9322a24f7def435a92159114e88b9d7c
I20260620 01:54:37.506624 9670 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 01:54:37.506805 9670 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 3 FOLLOWER]: Starting leader election (detected failure of leader f618f87e8d484910a4586badc3211eb0)
I20260620 01:54:37.506922 9670 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 3 FOLLOWER]: Advancing to term 4
I20260620 01:54:37.511511 9670 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:37.513155 9670 leader_election.cc:290] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [CANDIDATE]: Term 4 election: Requested vote from peers f618f87e8d484910a4586badc3211eb0 (127.20.1.190:42071), 9322a24f7def435a92159114e88b9d7c (127.20.1.189:37363)
I20260620 01:54:37.513763 8987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6edf4a403b194538857d38331fb4c16d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9322a24f7def435a92159114e88b9d7c"
I20260620 01:54:37.514464 9619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6edf4a403b194538857d38331fb4c16d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f618f87e8d484910a4586badc3211eb0"
I20260620 01:54:37.514827 9619 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 3 FOLLOWER]: Advancing to term 4
I20260620 01:54:37.529021 9619 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edf4a403b194538857d38331fb4c16d in term 4.
I20260620 01:54:37.530360 9061 leader_election.cc:304] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6edf4a403b194538857d38331fb4c16d, f618f87e8d484910a4586badc3211eb0; no voters: 9322a24f7def435a92159114e88b9d7c
I20260620 01:54:37.531093 9670 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 4 FOLLOWER]: Leader election won for term 4
I20260620 01:54:37.531886 9670 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [term 4 LEADER]: Becoming Leader. State: Replica: 6edf4a403b194538857d38331fb4c16d, State: Running, Role: LEADER
I20260620 01:54:37.532609 9670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:37.552583 9674 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f618f87e8d484910a4586badc3211eb0)
I20260620 01:54:37.553215 9674 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } }
I20260620 01:54:37.556452 9674 leader_election.cc:290] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f618f87e8d484910a4586badc3211eb0 (127.20.1.190:42071), 6edf4a403b194538857d38331fb4c16d (127.20.1.188:46173)
I20260620 01:54:37.565961 9672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6edf4a403b194538857d38331fb4c16d. Latest consensus state: current_term: 4 leader_uuid: "6edf4a403b194538857d38331fb4c16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:37.566447 9672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: This master's current role is: LEADER
I20260620 01:54:37.577440 9084 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9322a24f7def435a92159114e88b9d7c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6edf4a403b194538857d38331fb4c16d" is_pre_election: true
I20260620 01:54:37.580493 9677 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:54:37.585541 9677 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:54:37.586005 9619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9322a24f7def435a92159114e88b9d7c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f618f87e8d484910a4586badc3211eb0" is_pre_election: true
I20260620 01:54:37.586597 9619 raft_consensus.cc:2393] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9322a24f7def435a92159114e88b9d7c in current term 4: Already voted for candidate 6edf4a403b194538857d38331fb4c16d in this term.
I20260620 01:54:37.587579 9677 catalog_manager.cc:1294] Loaded cluster ID: 7c89cc2f428d4a57adaab580812f43ad
I20260620 01:54:37.587801 9677 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:54:37.587873 8963 leader_election.cc:304] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9322a24f7def435a92159114e88b9d7c; no voters: 6edf4a403b194538857d38331fb4c16d, f618f87e8d484910a4586badc3211eb0
I20260620 01:54:37.589077 9674 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 3 FOLLOWER]: Advancing to term 4
I20260620 01:54:37.590168 9677 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:54:37.591696 9677 catalog_manager.cc:6080] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d: Loaded TSK: 0
I20260620 01:54:37.593336 9677 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:54:37.593709 9674 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260620 01:54:37.796861 9668 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:37.797364 9668 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:37.797560 9668 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:37.808768 9668 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:37.809026 9668 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:54:37.821550 9668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:39993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=37433
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:37.823699 9668 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:37.825531 9668 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:37.839037 9682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:37.840014 9683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:37.840222 9685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:37.842276 9668 server_base.cc:1061] running on GCE node
I20260620 01:54:37.843205 9668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:37.844805 9668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:37.848338 9668 hybrid_clock.cc:648] HybridClock initialized: now 1781920477848171 us; error 93 us; skew 500 ppm
I20260620 01:54:37.849095 9668 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:37 dist-test-slave-wg14 krb5kdc[8830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920477, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:37.855197 9668 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:54:37.858232 9668 webserver.cc:533] Webserver started at http://127.20.1.129:37433/ using document root <none> and password file <none>
I20260620 01:54:37.859730 9668 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:37.860209 9668 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:37.868973 9668 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 01:54:37.874188 9692 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:37.877163 9668 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260620 01:54:37.877445 9668 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "cbec150f8e594225b4d792c3284bb950"
format_stamp: "Formatted at 2026-06-20 01:54:23 on dist-test-slave-wg14"
I20260620 01:54:37.878499 9668 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:37.899781 9668 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:37.908106 9668 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:37.908699 9668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:37.910748 9668 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:37.913341 9668 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:37.913478 9668 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:37.913623 9668 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:37.913709 9668 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:37.967875 9805 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:39993 every 8 connection(s)
I20260620 01:54:37.967902 9668 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:39993
I20260620 01:54:37.971655 9668 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:54:37.975504 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 9668
Jun 20 01:54:38 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920477, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:54:38 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920477, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:54:38 dist-test-slave-wg14 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920477, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:54:38.037045 9807 heartbeater.cc:344] Connected to a master server at 127.20.1.190:42071
I20260620 01:54:38.037268 9806 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46173
I20260620 01:54:38.037614 9807 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:38.037612 9806 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:38.037060 9808 heartbeater.cc:344] Connected to a master server at 127.20.1.189:37363
I20260620 01:54:38.038486 9808 heartbeater.cc:461] Registering TS with master...
I20260620 01:54:38.039400 9808 heartbeater.cc:507] Master 127.20.1.189:37363 requested a full tablet report, sending...
I20260620 01:54:38.039443 9806 heartbeater.cc:507] Master 127.20.1.188:46173 requested a full tablet report, sending...
I20260620 01:54:38.039489 9807 heartbeater.cc:507] Master 127.20.1.190:42071 requested a full tablet report, sending...
I20260620 01:54:38.043761 9074 ts_manager.cc:194] Re-registered known tserver with Master: cbec150f8e594225b4d792c3284bb950 (127.20.1.129:39993)
I20260620 01:54:38.043736 9609 ts_manager.cc:194] Registered new tserver with Master: cbec150f8e594225b4d792c3284bb950 (127.20.1.129:39993)
I20260620 01:54:38.044030 8976 ts_manager.cc:194] Re-registered known tserver with Master: cbec150f8e594225b4d792c3284bb950 (127.20.1.129:39993)
I20260620 01:54:38.046715 9074 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:48453
I20260620 01:54:38.055395 9619 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [term 4 FOLLOWER]: Refusing update from remote peer 6edf4a403b194538857d38331fb4c16d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260620 01:54:38.056588 9672 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 01:54:38.062254 9817 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:38.056787 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:38986 (local address 127.20.1.129:39993)
0620 01:54:38.057707 (+ 920us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:38.057719 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:38.058731 (+ 1012us) server_negotiation.cc:408] Connection header received
0620 01:54:38.058915 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:38.058924 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:38.059266 (+ 342us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:38.059476 (+ 210us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:38.060709 (+ 1233us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38986: BlockingRecv error: recv got EOF from 127.0.0.1:38986 (error 108)
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":460,"threads_started":1}
I20260620 01:54:38.064019 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 9668
I20260620 01:54:38.065253 9818 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6edf4a403b194538857d38331fb4c16d. Latest consensus state: current_term: 4 leader_uuid: "6edf4a403b194538857d38331fb4c16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:38.065704 9818 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:54:38.070096 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 9668: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:39993: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:54:38.070420 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 9668
I20260620 01:54:38.076900 9818 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "6edf4a403b194538857d38331fb4c16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:38.077482 9818 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618f87e8d484910a4586badc3211eb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:38.083771 8987 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [term 4 FOLLOWER]: Refusing update from remote peer 6edf4a403b194538857d38331fb4c16d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260620 01:54:38.085520 9672 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 01:54:38.092005 9672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "6edf4a403b194538857d38331fb4c16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:38.092298 9672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: This master's current role is: LEADER
I20260620 01:54:38.100435 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:39993
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--webserver_port=37433
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:38.100337 9826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "6edf4a403b194538857d38331fb4c16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:38.101874 9826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:54:38.101259 9674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6edf4a403b194538857d38331fb4c16d. Latest consensus state: current_term: 4 leader_uuid: "6edf4a403b194538857d38331fb4c16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:38.102036 9674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9322a24f7def435a92159114e88b9d7c [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:54:38.649360 9827 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:38.649827 9827 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:38.649981 9827 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:54:38.650094 9827 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:38.661309 9827 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:38.661800 9827 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:54:38.679908 9827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:39993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=37433
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:38.682859 9827 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:38.685729 9827 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260620 01:54:38.690495 9823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "6edf4a403b194538857d38331fb4c16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f618f87e8d484910a4586badc3211eb0" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42071 } } peers { permanent_uuid: "9322a24f7def435a92159114e88b9d7c" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37363 } } peers { permanent_uuid: "6edf4a403b194538857d38331fb4c16d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46173 } } }
I20260620 01:54:38.691216 9823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6edf4a403b194538857d38331fb4c16d [sys.catalog]: This master's current role is: LEADER
W20260620 01:54:38.711652 9836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:38.711913 9838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:38.712668 9835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:38.713274 9827 server_base.cc:1061] running on GCE node
I20260620 01:54:38.714290 9827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:38.715966 9827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:38.719574 9827 hybrid_clock.cc:648] HybridClock initialized: now 1781920478719294 us; error 139 us; skew 500 ppm
I20260620 01:54:38.720693 9827 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:38 dist-test-slave-wg14 krb5kdc[8830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920478, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:38.730417 9827 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:54:38.733110 9827 webserver.cc:533] Webserver started at http://127.20.1.129:37433/ using document root <none> and password file <none>
I20260620 01:54:38.734256 9827 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:38.734388 9827 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:38.742530 9827 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:54:38.747381 9845 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:38.750612 9827 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.001s sys 0.002s
I20260620 01:54:38.751022 9827 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "cbec150f8e594225b4d792c3284bb950"
format_stamp: "Formatted at 2026-06-20 01:54:23 on dist-test-slave-wg14"
I20260620 01:54:38.752529 9827 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:38.790295 9827 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:38.797101 9827 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:38.797442 9827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:38.798879 9827 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:38.801543 9827 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:38.801703 9827 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:38.801858 9827 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:38.801915 9827 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:54:38.802532 9827 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:54:38.802621 9827 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:54:38.802845 9827 fault_injection.cc:59] injecting fault for kudu (pid 9827): FLAGS_fault_before_start (process will exit)
I20260620 01:54:38.827514 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:39993
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--webserver_port=37433
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:54:39.234858 9564 heartbeater.cc:499] Master 127.20.1.188:46173 was elected leader, sending a full tablet report...
W20260620 01:54:39.316771 9853 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:54:39.317198 9853 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:54:39.317343 9853 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:54:39.328485 9853 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:54:39.328897 9853 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:54:39.343364 9853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39379,127.20.1.149:46737,127.20.1.150:33371,127.20.1.151:34843,127.20.1.152:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:39993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=37433
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:42071,127.20.1.189:37363,127.20.1.188:46173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:54:39.345996 9853 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:54:39.348688 9853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:54:39.364315 9859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:39.364352 9858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:54:39.364455 9861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:54:39.367264 9853 server_base.cc:1061] running on GCE node
I20260620 01:54:39.368230 9853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:54:39.369899 9853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:54:39.373607 9853 hybrid_clock.cc:648] HybridClock initialized: now 1781920479373264 us; error 152 us; skew 500 ppm
I20260620 01:54:39.374544 9853 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:54:39 dist-test-slave-wg14 krb5kdc[8830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920479, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:54:39.380450 9853 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:54:39.383267 9853 webserver.cc:533] Webserver started at http://127.20.1.129:37433/ using document root <none> and password file <none>
I20260620 01:54:39.384392 9853 fs_manager.cc:362] Metadata directory not provided
I20260620 01:54:39.384580 9853 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:54:39.392907 9853 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260620 01:54:39.397712 9868 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:39.400736 9853 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260620 01:54:39.401053 9853 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "cbec150f8e594225b4d792c3284bb950"
format_stamp: "Formatted at 2026-06-20 01:54:23 on dist-test-slave-wg14"
I20260620 01:54:39.402137 9853 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:54:39.414397 9420 heartbeater.cc:499] Master 127.20.1.188:46173 was elected leader, sending a full tablet report...
I20260620 01:54:39.437458 9853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:54:39.443418 9853 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:54:39.443805 9853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:54:39.445241 9853 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:54:39.447813 9853 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:54:39.447973 9853 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:39.448119 9853 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:54:39.448203 9853 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:54:39.493525 9853 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:39993
I20260620 01:54:39.493595 9981 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:39993 every 8 connection(s)
I20260620 01:54:39.495674 9853 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:54:39.497663 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 9853
I20260620 01:54:39.498085 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 9853
W20260620 01:54:39.518754 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 9853: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:39993: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:54:39.519014 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 9853
I20260620 01:54:39.541926 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 9291
W20260620 01:54:39.542833 9987 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:39.501528 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:59733 (local address 127.20.1.188:46173)
0620 01:54:39.511222 (+ 9694us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:39.511229 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:39.512826 (+ 1597us) server_negotiation.cc:408] Connection header received
0620 01:54:39.520694 (+ 7868us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:39.520704 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:39.520837 (+ 133us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:39.521024 (+ 187us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:39.541535 (+ 20511us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:59733: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":9507,"thread_start_us":9038,"threads_started":1}
W20260620 01:54:39.542933 9991 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:39.514100 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:47257 (local address 127.20.1.190:42071)
0620 01:54:39.529418 (+ 15318us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:39.529426 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:39.541847 (+ 12421us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:47257: BlockingRecv error: recv got EOF from 127.20.1.129:47257 (error 108)
Metrics: {"server-negotiator.queue_time_us":15144,"thread_start_us":15034,"threads_started":1}
W20260620 01:54:39.542941 9990 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:39.513868 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:43867 (local address 127.20.1.189:37363)
0620 01:54:39.524463 (+ 10595us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:39.524469 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:39.541801 (+ 17332us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:43867: BlockingRecv error: recv got EOF from 127.20.1.129:43867 (error 108)
Metrics: {"server-negotiator.queue_time_us":10421,"thread_start_us":10242,"threads_started":1}
W20260620 01:54:39.546352 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 9291: Not authorized: Client connection negotiation failed: client connection to 127.20.1.130:35511: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:54:39.546562 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 9291
I20260620 01:54:39.562216 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 9435
W20260620 01:54:39.566355 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 9435: Not authorized: Client connection negotiation failed: client connection to 127.20.1.131:41279: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:54:39.566607 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 9435
I20260620 01:54:39.584249 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 9575
W20260620 01:54:39.587569 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 9575: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:42071: server requires authentication, but client does not have Kerberos credentials available
W20260620 01:54:39.587694 9991 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:54:39.585873 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.20.1.190:42071)
0620 01:54:39.586121 (+ 248us) server_negotiation.cc:207] Beginning negotiation
0620 01:54:39.586126 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0620 01:54:39.586144 (+ 18us) server_negotiation.cc:408] Connection header received
0620 01:54:39.586208 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:54:39.586213 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:54:39.586288 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:54:39.586399 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:54:39.587399 (+ 1000us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49082: BlockingRecv error: recv got EOF from 127.0.0.1:49082 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 01:54:39.587788 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 9575
I20260620 01:54:39.612252 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 8944
I20260620 01:54:39.614240 9007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:54:39.751176 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 8944
I20260620 01:54:39.778895 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 9035
I20260620 01:54:39.781145 9104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:54:39.913911 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 9035
2026-06-20T01:54:39Z chronyd exiting
2026-06-20T01:54:39Z chronyd exiting
2026-06-20T01:54:39Z chronyd exiting
2026-06-20T01:54:39Z chronyd exiting
2026-06-20T01:54:39Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/8 (23209 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/9
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 01:54:40 dist-test-slave-wg14 krb5kdc[9999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 01:54:40 dist-test-slave-wg14 krb5kdc[9999](info): set up 2 sockets
Jun 20 01:54:40 dist-test-slave-wg14 krb5kdc[9999](info): commencing operation
krb5kdc: starting...
W20260620 01:54:42.091661 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.083s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 01:54:42 dist-test-slave-wg14 krb5kdc[9999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf
2026-06-20 01:54:52: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Jun 20 01:54:55 dist-test-slave-wg14 krb5kdc[9999](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920495, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 01:55:02 dist-test-slave-wg14 krb5kdc[9999](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920502, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 01:55:02 dist-test-slave-wg14 krb5kdc[9999](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920502, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:55:05.141634 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:34393
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:55:05.589215 10298 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:05.589681 10298 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:05.589843 10298 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:05.600605 10298 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:55:05.600775 10298 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:05.600860 10298 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:55:05.600931 10298 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:55:05.613646 10298 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:34393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:05.615900 10298 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:05.617784 10298 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:05.630517 10303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:05.630478 10304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:05.630565 10306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:05.631343 10298 server_base.cc:1061] running on GCE node
I20260620 01:55:05.633147 10298 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:05.633404 10298 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:05.633528 10298 hybrid_clock.cc:648] HybridClock initialized: now 1781920505633500 us; error 0 us; skew 500 ppm
I20260620 01:55:05.634001 10298 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:05 dist-test-slave-wg14 krb5kdc[9999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920505, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:05.638880 10298 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:55:05.642326 10298 webserver.cc:533] Webserver started at http://127.20.1.190:38457/ using document root <none> and password file <none>
I20260620 01:55:05.643666 10298 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:05.643833 10298 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:05.644244 10298 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:55:05.648279 10298 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "e939ebcbbd4c4fc3b4da8246c17c2135"
format_stamp: "Formatted at 2026-06-20 01:55:05 on dist-test-slave-wg14"
I20260620 01:55:05.649608 10298 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "e939ebcbbd4c4fc3b4da8246c17c2135"
format_stamp: "Formatted at 2026-06-20 01:55:05 on dist-test-slave-wg14"
I20260620 01:55:05.657927 10298 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260620 01:55:05.663923 10312 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:05.666519 10298 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.002s
I20260620 01:55:05.666795 10298 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "e939ebcbbd4c4fc3b4da8246c17c2135"
format_stamp: "Formatted at 2026-06-20 01:55:05 on dist-test-slave-wg14"
I20260620 01:55:05.667246 10298 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:05.702669 10298 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:05.713335 10298 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:05.713735 10298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:05.742769 10298 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:34393
I20260620 01:55:05.742784 10364 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:34393 every 8 connection(s)
I20260620 01:55:05.744936 10298 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:55:05.748402 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 10298
I20260620 01:55:05.748637 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:55:05.752372 10365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:55:05.762320 10365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.189@KRBTEST.COM" created.
Jun 20 01:55:05 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920505, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:55:05.799732 10365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:55:05.801913 10315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36651: connect: Connection refused (error 111)
W20260620 01:55:05.805091 10365 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36651: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36651: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.189@KRBTEST.COM" created.
I20260620 01:55:05.854722 10365 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } attempt: 1
W20260620 01:55:05.857496 10365 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36651: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36651: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:55:05.881012 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:36651
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:55:05.925905 10365 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } attempt: 2
W20260620 01:55:05.928572 10365 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36651: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36651: connect: Connection refused (error 111)
I20260620 01:55:06.020004 10365 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } attempt: 3
W20260620 01:55:06.022807 10365 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36651: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36651: connect: Connection refused (error 111)
I20260620 01:55:06.166641 10365 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } attempt: 4
W20260620 01:55:06.169624 10365 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36651: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36651: connect: Connection refused (error 111)
W20260620 01:55:06.332281 10374 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:06.332785 10374 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:06.333115 10374 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:06.344179 10374 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:55:06.344360 10374 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:06.344425 10374 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:55:06.344470 10374 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:55:06.359360 10374 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:36651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:06.361666 10374 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:06.363502 10374 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:06.375520 10380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:06.375466 10379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:06.375519 10382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:06.376753 10374 server_base.cc:1061] running on GCE node
I20260620 01:55:06.377951 10374 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:06.378209 10374 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:06.378316 10374 hybrid_clock.cc:648] HybridClock initialized: now 1781920506378289 us; error 0 us; skew 500 ppm
I20260620 01:55:06.378754 10374 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:06 dist-test-slave-wg14 krb5kdc[9999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920506, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:06.384613 10374 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 01:55:06.387640 10374 webserver.cc:533] Webserver started at http://127.20.1.189:35201/ using document root <none> and password file <none>
I20260620 01:55:06.388741 10374 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:06.388895 10374 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:06.389263 10374 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:55:06.392886 10374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "670ce53faa5041b5aebd1e588d7b1124"
format_stamp: "Formatted at 2026-06-20 01:55:06 on dist-test-slave-wg14"
I20260620 01:55:06.394217 10374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "670ce53faa5041b5aebd1e588d7b1124"
format_stamp: "Formatted at 2026-06-20 01:55:06 on dist-test-slave-wg14"
I20260620 01:55:06.402055 10374 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260620 01:55:06.407167 10388 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:06.409624 10374 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260620 01:55:06.409929 10374 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "670ce53faa5041b5aebd1e588d7b1124"
format_stamp: "Formatted at 2026-06-20 01:55:06 on dist-test-slave-wg14"
I20260620 01:55:06.410187 10374 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:06.444701 10374 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:06.450366 10374 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:06.450768 10374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:06.469149 10365 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } attempt: 5
W20260620 01:55:06.472494 10365 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36651: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36651: connect: Connection refused (error 111)
I20260620 01:55:06.478223 10374 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:36651
I20260620 01:55:06.478262 10440 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:36651 every 8 connection(s)
I20260620 01:55:06.480175 10374 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:55:06.486387 10441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:55:06.487032 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 10374
I20260620 01:55:06.487494 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:55:06.495755 10441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.188@KRBTEST.COM" created.
Jun 20 01:55:06 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920506, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:55:06.532416 10441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:55:06 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920506, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:55:06.555932 10441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:55:06.558477 10389 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39809: connect: Connection refused (error 111)
W20260620 01:55:06.561570 10441 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:39809: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39809: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.188@KRBTEST.COM" created.
I20260620 01:55:06.611302 10441 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } attempt: 1
W20260620 01:55:06.615392 10441 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:39809: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39809: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:55:06.637236 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:39809
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:55:06.683774 10441 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } attempt: 2
W20260620 01:55:06.687304 10441 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:39809: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39809: connect: Connection refused (error 111)
I20260620 01:55:06.778743 10441 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } attempt: 3
W20260620 01:55:06.781999 10441 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:39809: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39809: connect: Connection refused (error 111)
I20260620 01:55:06.925369 10441 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } attempt: 4
W20260620 01:55:06.928821 10441 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:39809: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39809: connect: Connection refused (error 111)
I20260620 01:55:07.019860 10365 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } attempt: 6
Jun 20 01:55:07 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920505, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:55:07.042709 10365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:55:07.047168 10365 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:39809: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39809: connect: Connection refused (error 111)
I20260620 01:55:07.099573 10365 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } attempt: 1
W20260620 01:55:07.102643 10365 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:39809: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39809: connect: Connection refused (error 111)
W20260620 01:55:07.158186 10451 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:07.158599 10451 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:07.158756 10451 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:07.175433 10451 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:55:07.175587 10451 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:07.175654 10451 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:55:07.175709 10451 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:55:07.176992 10365 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } attempt: 2
W20260620 01:55:07.179559 10365 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:39809: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39809: connect: Connection refused (error 111)
I20260620 01:55:07.190162 10451 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:39809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:07.193807 10451 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:07.195793 10451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:07.208973 10459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:07.208134 10458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:07.212906 10461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:07.213828 10451 server_base.cc:1061] running on GCE node
I20260620 01:55:07.215698 10451 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:07.216158 10451 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:07.216470 10451 hybrid_clock.cc:648] HybridClock initialized: now 1781920507216436 us; error 0 us; skew 500 ppm
I20260620 01:55:07.217305 10451 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:07 dist-test-slave-wg14 krb5kdc[9999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920507, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:07.223840 10451 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 01:55:07.226922 10451 webserver.cc:533] Webserver started at http://127.20.1.188:37067/ using document root <none> and password file <none>
I20260620 01:55:07.227954 10451 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:07.228149 10451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:07.228185 10441 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } attempt: 5
I20260620 01:55:07.228806 10451 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
W20260620 01:55:07.233844 10441 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:39809: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39809: connect: Connection refused (error 111)
I20260620 01:55:07.234261 10451 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "a892964204a44e5a8efd106641f0714a"
format_stamp: "Formatted at 2026-06-20 01:55:07 on dist-test-slave-wg14"
I20260620 01:55:07.236720 10451 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "a892964204a44e5a8efd106641f0714a"
format_stamp: "Formatted at 2026-06-20 01:55:07 on dist-test-slave-wg14"
I20260620 01:55:07.247625 10451 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s
I20260620 01:55:07.253930 10467 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:07.256956 10451 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:55:07.257455 10451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "a892964204a44e5a8efd106641f0714a"
format_stamp: "Formatted at 2026-06-20 01:55:07 on dist-test-slave-wg14"
I20260620 01:55:07.257783 10451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:07.283883 10451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:07.290140 10451 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:07.290460 10451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:07.292886 10365 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } attempt: 3
W20260620 01:55:07.295604 10365 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:39809: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39809: connect: Connection refused (error 111)
I20260620 01:55:07.322960 10451 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:39809
I20260620 01:55:07.323052 10519 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:39809 every 8 connection(s)
I20260620 01:55:07.325336 10451 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:55:07.327021 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 10451
I20260620 01:55:07.327481 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:55:07.334854 10520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 01:55:07 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:55:07.346720 10520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:55:07 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:55:07 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920507, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:55:07.383849 10520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:55:07 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920507, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:55:07 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:55:07.403560 10520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:55:07 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920507, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:55:07.429848 10520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: Bootstrap starting.
I20260620 01:55:07.435091 10520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: Neither blocks nor log segments found. Creating new log.
I20260620 01:55:07.436678 10520 log.cc:826] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: Log is configured to *not* fsync() on all Append() calls
I20260620 01:55:07.440697 10520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: No bootstrap required, opened a new log
I20260620 01:55:07.444993 10365 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } attempt: 4
I20260620 01:55:07.449280 10520 raft_consensus.cc:359] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:07.449995 10520 raft_consensus.cc:385] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:55:07.450228 10520 raft_consensus.cc:740] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a892964204a44e5a8efd106641f0714a, State: Initialized, Role: FOLLOWER
I20260620 01:55:07.451236 10520 consensus_queue.cc:260] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:07.452687 10528 sys_catalog.cc:455] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:07.453019 10528 sys_catalog.cc:458] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:07.454506 10520 sys_catalog.cc:565] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 01:55:07 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920505, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
W20260620 01:55:07.469632 10539 catalog_manager.cc:1593] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:55:07.469831 10539 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:55:07.465853 10451 master_runner.cc:187] Error getting master registration for 127.20.1.190:34393: OK, instance_id { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" instance_seqno: 1781920505734870 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:55:07.473731 10451 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:55:07.475905 10365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Bootstrap starting.
I20260620 01:55:07.482851 10365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Neither blocks nor log segments found. Creating new log.
I20260620 01:55:07.485085 10365 log.cc:826] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Log is configured to *not* fsync() on all Append() calls
I20260620 01:55:07.490289 10365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: No bootstrap required, opened a new log
I20260620 01:55:07.498588 10365 raft_consensus.cc:359] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:07.499259 10365 raft_consensus.cc:385] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:55:07.499455 10365 raft_consensus.cc:740] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e939ebcbbd4c4fc3b4da8246c17c2135, State: Initialized, Role: FOLLOWER
I20260620 01:55:07.500420 10365 consensus_queue.cc:260] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:07.502072 10542 sys_catalog.cc:455] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:07.502689 10542 sys_catalog.cc:458] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:07.503778 10365 sys_catalog.cc:565] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:55:07.519261 10553 catalog_manager.cc:1593] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:55:07.519719 10553 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:55:07.519620 10298 master_runner.cc:187] Error getting master registration for 127.20.1.189:36651: OK, instance_id { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" instance_seqno: 1781920506470962 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:55:07.528199 10298 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:55:07.544123 10473 tablet.cc:2406] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:55:07.712154 10542 raft_consensus.cc:493] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:55:07.712565 10542 raft_consensus.cc:515] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:07.716616 10542 leader_election.cc:290] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 670ce53faa5041b5aebd1e588d7b1124 (127.20.1.189:36651), a892964204a44e5a8efd106641f0714a (127.20.1.188:39809)
I20260620 01:55:07.716526 10494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a892964204a44e5a8efd106641f0714a" is_pre_election: true
I20260620 01:55:07.717470 10494 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e939ebcbbd4c4fc3b4da8246c17c2135 in term 0.
I20260620 01:55:07.718588 10313 leader_election.cc:304] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a892964204a44e5a8efd106641f0714a, e939ebcbbd4c4fc3b4da8246c17c2135; no voters:
W20260620 01:55:07.719153 10318 tablet.cc:2406] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:55:07.719368 10542 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:55:07.719600 10542 raft_consensus.cc:493] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:55:07.719733 10542 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:55:07.723594 10542 raft_consensus.cc:515] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:07.724795 10542 leader_election.cc:290] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [CANDIDATE]: Term 1 election: Requested vote from peers 670ce53faa5041b5aebd1e588d7b1124 (127.20.1.189:36651), a892964204a44e5a8efd106641f0714a (127.20.1.188:39809)
I20260620 01:55:07.726502 10494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a892964204a44e5a8efd106641f0714a"
I20260620 01:55:07.726815 10494 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:55:07.731024 10494 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e939ebcbbd4c4fc3b4da8246c17c2135 in term 1.
I20260620 01:55:07.732002 10313 leader_election.cc:304] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a892964204a44e5a8efd106641f0714a, e939ebcbbd4c4fc3b4da8246c17c2135; no voters:
I20260620 01:55:07.732621 10542 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:55:07.733485 10542 raft_consensus.cc:697] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 1 LEADER]: Becoming Leader. State: Replica: e939ebcbbd4c4fc3b4da8246c17c2135, State: Running, Role: LEADER
I20260620 01:55:07.734278 10542 consensus_queue.cc:237] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:07.744237 10555 sys_catalog.cc:455] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e939ebcbbd4c4fc3b4da8246c17c2135. Latest consensus state: current_term: 1 leader_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:07.744693 10555 sys_catalog.cc:458] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: This master's current role is: LEADER
I20260620 01:55:07.754622 10557 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:55:07.761961 10557 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:55:07.776564 10494 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 1 FOLLOWER]: Refusing update from remote peer e939ebcbbd4c4fc3b4da8246c17c2135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:55:07.778800 10555 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:55:07.781167 10441 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } attempt: 6
I20260620 01:55:07.811517 10528 sys_catalog.cc:455] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e939ebcbbd4c4fc3b4da8246c17c2135. Latest consensus state: current_term: 1 leader_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:07.812057 10528 sys_catalog.cc:458] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:07.819814 10542 sys_catalog.cc:455] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:07.820676 10542 sys_catalog.cc:458] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: This master's current role is: LEADER
I20260620 01:55:07.828029 10557 catalog_manager.cc:1382] Generated new cluster ID: a361767a6b444190b4b4e4a25e4ef9c4
I20260620 01:55:07.829072 10528 sys_catalog.cc:455] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:07.829416 10528 sys_catalog.cc:458] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 01:55:07 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920506, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:55:07.832437 10557 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:55:07.860110 10441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124: Bootstrap starting.
I20260620 01:55:07.866597 10441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124: Neither blocks nor log segments found. Creating new log.
I20260620 01:55:07.868907 10441 log.cc:826] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124: Log is configured to *not* fsync() on all Append() calls
I20260620 01:55:07.883896 10557 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:55:07.889917 10557 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:55:07.890076 10441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124: No bootstrap required, opened a new log
I20260620 01:55:07.902784 10441 raft_consensus.cc:359] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:07.904210 10441 raft_consensus.cc:385] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:55:07.904489 10441 raft_consensus.cc:740] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 670ce53faa5041b5aebd1e588d7b1124, State: Initialized, Role: FOLLOWER
I20260620 01:55:07.905750 10441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:07.918992 10565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:07.922339 10565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:07.922093 10441 sys_catalog.cc:565] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:55:07.922658 10413 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:55:07.927263 10416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "670ce53faa5041b5aebd1e588d7b1124" is_pre_election: true
I20260620 01:55:07.929606 10415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "670ce53faa5041b5aebd1e588d7b1124"
I20260620 01:55:07.931224 10413 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 1 FOLLOWER]: Refusing update from remote peer e939ebcbbd4c4fc3b4da8246c17c2135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:55:07.932720 10542 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 01:55:07.955618 10394 tablet.cc:2406] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:55:07.965646 10557 catalog_manager.cc:6069] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Generated new TSK 0
I20260620 01:55:07.967566 10557 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:55:08.014827 10570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:08.015300 10570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:08.015781 10565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e939ebcbbd4c4fc3b4da8246c17c2135. Latest consensus state: current_term: 1 leader_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:08.016029 10565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:08.016058 10555 sys_catalog.cc:455] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:08.016332 10555 sys_catalog.cc:458] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: This master's current role is: LEADER
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:55:08.096758 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.111s user 0.000s sys 0.023s
I20260620 01:55:08.119423 10583 catalog_manager.cc:1294] Loaded cluster ID: a361767a6b444190b4b4e4a25e4ef9c4
I20260620 01:55:08.123709 10583 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124: loading cluster ID for follower catalog manager: success
I20260620 01:55:08.129566 10583 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124: acquiring CA information for follower catalog manager: success
I20260620 01:55:08.138986 10583 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:55:08.202271 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.105s user 0.000s sys 0.024s
I20260620 01:55:08.203488 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:55:08.473066 10539 catalog_manager.cc:1294] Loaded cluster ID: a361767a6b444190b4b4e4a25e4ef9c4
I20260620 01:55:08.473217 10539 catalog_manager.cc:1587] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: loading cluster ID for follower catalog manager: success
I20260620 01:55:08.479017 10539 catalog_manager.cc:1609] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: acquiring CA information for follower catalog manager: success
I20260620 01:55:08.483322 10539 catalog_manager.cc:1637] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 01:55:08.676117 10587 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:08.676533 10587 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:08.676697 10587 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:08.688277 10587 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:08.688578 10587 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:55:08.703001 10587 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:08.705004 10587 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:08.707381 10587 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:08.721853 10593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:08.722431 10594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:08.723505 10596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:08.724673 10587 server_base.cc:1061] running on GCE node
I20260620 01:55:08.726202 10587 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:08.726444 10587 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:08.726550 10587 hybrid_clock.cc:648] HybridClock initialized: now 1781920508726520 us; error 0 us; skew 500 ppm
I20260620 01:55:08.727035 10587 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:08 dist-test-slave-wg14 krb5kdc[9999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920508, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:08.731788 10587 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:55:08.734368 10587 webserver.cc:533] Webserver started at http://127.20.1.129:40645/ using document root <none> and password file <none>
I20260620 01:55:08.735451 10587 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:08.735610 10587 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:08.736020 10587 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:55:08.739079 10587 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "39486b58a61649dfababf1d4b5322a97"
format_stamp: "Formatted at 2026-06-20 01:55:08 on dist-test-slave-wg14"
I20260620 01:55:08.740474 10587 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "39486b58a61649dfababf1d4b5322a97"
format_stamp: "Formatted at 2026-06-20 01:55:08 on dist-test-slave-wg14"
I20260620 01:55:08.750238 10587 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260620 01:55:08.756723 10603 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:08.759621 10587 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:55:08.759898 10587 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "39486b58a61649dfababf1d4b5322a97"
format_stamp: "Formatted at 2026-06-20 01:55:08 on dist-test-slave-wg14"
I20260620 01:55:08.760214 10587 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:08.790758 10587 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:08.796418 10587 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:08.796834 10587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:08.798075 10587 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:55:08.800362 10587 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:55:08.800472 10587 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:08.800598 10587 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:55:08.800683 10587 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:08.845256 10587 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35445
I20260620 01:55:08.845352 10717 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35445 every 8 connection(s)
I20260620 01:55:08.847108 10587 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:55:08.853461 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 10587
I20260620 01:55:08.853782 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 01:55:08 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920508, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:55:08 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920508, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:55:08 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920508, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:55:08.907835 10719 heartbeater.cc:344] Connected to a master server at 127.20.1.188:39809
I20260620 01:55:08.908107 10718 heartbeater.cc:344] Connected to a master server at 127.20.1.190:34393
WARNING: no policy specified for kudu/127.20.1.130@KRBTEST.COM; defaulting to no policy
I20260620 01:55:08.908371 10719 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:08.909179 10720 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36651
I20260620 01:55:08.909174 10718 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:08.909476 10720 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:08.910101 10719 heartbeater.cc:507] Master 127.20.1.188:39809 requested a full tablet report, sending...
I20260620 01:55:08.910173 10718 heartbeater.cc:507] Master 127.20.1.190:34393 requested a full tablet report, sending...
I20260620 01:55:08.910212 10720 heartbeater.cc:507] Master 127.20.1.189:36651 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.130@KRBTEST.COM" created.
I20260620 01:55:08.913321 10330 ts_manager.cc:194] Registered new tserver with Master: 39486b58a61649dfababf1d4b5322a97 (127.20.1.129:35445)
I20260620 01:55:08.916142 10330 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:57171
I20260620 01:55:08.917496 10484 ts_manager.cc:194] Registered new tserver with Master: 39486b58a61649dfababf1d4b5322a97 (127.20.1.129:35445)
I20260620 01:55:08.917915 10405 ts_manager.cc:194] Registered new tserver with Master: 39486b58a61649dfababf1d4b5322a97 (127.20.1.129:35445)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:55:09.037534 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:55:09.511736 10732 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:09.512312 10732 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:09.512485 10732 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:09.522454 10732 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:09.522737 10732 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:55:09.537029 10732 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:09.539073 10732 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:09.540829 10732 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:09.554104 10737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:09.554215 10738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:09.555483 10740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:09.556761 10732 server_base.cc:1061] running on GCE node
I20260620 01:55:09.557638 10732 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:09.557852 10732 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:09.557968 10732 hybrid_clock.cc:648] HybridClock initialized: now 1781920509557939 us; error 0 us; skew 500 ppm
I20260620 01:55:09.558384 10732 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:09 dist-test-slave-wg14 krb5kdc[9999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920509, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:09.564224 10732 init.cc:377] Logged in from keytab as kudu/127.20.1.130@KRBTEST.COM (short username kudu)
I20260620 01:55:09.569142 10732 webserver.cc:533] Webserver started at http://127.20.1.130:39811/ using document root <none> and password file <none>
I20260620 01:55:09.570415 10732 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:09.570566 10732 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:09.571426 10732 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:55:09.575055 10732 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/data/instance:
uuid: "f5b050223e074795b659b2c3b4273459"
format_stamp: "Formatted at 2026-06-20 01:55:09 on dist-test-slave-wg14"
I20260620 01:55:09.576507 10732 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance:
uuid: "f5b050223e074795b659b2c3b4273459"
format_stamp: "Formatted at 2026-06-20 01:55:09 on dist-test-slave-wg14"
I20260620 01:55:09.585377 10732 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.008s sys 0.001s
I20260620 01:55:09.591398 10746 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:09.593941 10732 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:55:09.594259 10732 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "f5b050223e074795b659b2c3b4273459"
format_stamp: "Formatted at 2026-06-20 01:55:09 on dist-test-slave-wg14"
I20260620 01:55:09.594519 10732 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:09.629037 10732 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:09.634601 10732 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:09.634958 10732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:09.636464 10732 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:55:09.638880 10732 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:55:09.639025 10732 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:09.639175 10732 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:55:09.639245 10732 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 01:55:09.683276 10732 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:41181
I20260620 01:55:09.683640 10859 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:41181 every 8 connection(s)
I20260620 01:55:09.685236 10732 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:55:09.695592 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 10732
I20260620 01:55:09.695839 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
Jun 20 01:55:09 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920509, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:55:09 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920509, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:55:09 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920509, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:55:09.735355 10860 heartbeater.cc:344] Connected to a master server at 127.20.1.190:34393
I20260620 01:55:09.735423 10861 heartbeater.cc:344] Connected to a master server at 127.20.1.188:39809
I20260620 01:55:09.735605 10862 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36651
I20260620 01:55:09.736125 10860 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:09.736107 10861 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:09.736107 10862 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:09.737907 10862 heartbeater.cc:507] Master 127.20.1.189:36651 requested a full tablet report, sending...
I20260620 01:55:09.737943 10861 heartbeater.cc:507] Master 127.20.1.188:39809 requested a full tablet report, sending...
I20260620 01:55:09.737972 10860 heartbeater.cc:507] Master 127.20.1.190:34393 requested a full tablet report, sending...
I20260620 01:55:09.740573 10330 ts_manager.cc:194] Registered new tserver with Master: f5b050223e074795b659b2c3b4273459 (127.20.1.130:41181)
I20260620 01:55:09.740691 10405 ts_manager.cc:194] Registered new tserver with Master: f5b050223e074795b659b2c3b4273459 (127.20.1.130:41181)
I20260620 01:55:09.741778 10330 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:58261
I20260620 01:55:09.742046 10484 ts_manager.cc:194] Registered new tserver with Master: f5b050223e074795b659b2c3b4273459 (127.20.1.130:41181)
WARNING: no policy specified for kudu/127.20.1.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:55:09.847910 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:55:09.921245 10718 heartbeater.cc:499] Master 127.20.1.190:34393 was elected leader, sending a full tablet report...
W20260620 01:55:10.410101 10874 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:10.410991 10874 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:10.411445 10874 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:10.428030 10874 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:10.428318 10874 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:55:10.444789 10874 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:10.446929 10874 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:10.449064 10874 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:10.464897 10879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:10.465042 10880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:10.467777 10882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:10.468601 10874 server_base.cc:1061] running on GCE node
I20260620 01:55:10.472025 10874 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:10.472344 10874 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:10.472697 10874 hybrid_clock.cc:648] HybridClock initialized: now 1781920510472628 us; error 0 us; skew 500 ppm
I20260620 01:55:10.473208 10874 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:10 dist-test-slave-wg14 krb5kdc[9999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920510, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:10.478788 10874 init.cc:377] Logged in from keytab as kudu/127.20.1.131@KRBTEST.COM (short username kudu)
I20260620 01:55:10.481760 10874 webserver.cc:533] Webserver started at http://127.20.1.131:46631/ using document root <none> and password file <none>
I20260620 01:55:10.483363 10874 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:10.483578 10874 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:10.484081 10874 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:55:10.488826 10874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/data/instance:
uuid: "0652566484b24e5cb7acc917520c310b"
format_stamp: "Formatted at 2026-06-20 01:55:10 on dist-test-slave-wg14"
I20260620 01:55:10.490056 10874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance:
uuid: "0652566484b24e5cb7acc917520c310b"
format_stamp: "Formatted at 2026-06-20 01:55:10 on dist-test-slave-wg14"
I20260620 01:55:10.498909 10874 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260620 01:55:10.505081 10888 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:10.508652 10874 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
I20260620 01:55:10.509109 10874 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "0652566484b24e5cb7acc917520c310b"
format_stamp: "Formatted at 2026-06-20 01:55:10 on dist-test-slave-wg14"
I20260620 01:55:10.509459 10874 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:10.542927 10874 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:10.549732 10874 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:10.550160 10874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:10.551657 10874 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:55:10.554502 10874 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:55:10.554651 10874 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:10.554809 10874 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:55:10.554872 10874 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:10.624136 10874 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:35727
I20260620 01:55:10.624408 11001 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:35727 every 8 connection(s)
I20260620 01:55:10.625934 10874 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:55:10.635656 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 10874
I20260620 01:55:10.637454 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
Jun 20 01:55:10 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920510, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:55:10 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920510, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:55:10 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920510, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:55:10.684962 11003 heartbeater.cc:344] Connected to a master server at 127.20.1.188:39809
I20260620 01:55:10.685108 11002 heartbeater.cc:344] Connected to a master server at 127.20.1.190:34393
I20260620 01:55:10.685601 11003 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:10.685642 11002 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:10.688427 11002 heartbeater.cc:507] Master 127.20.1.190:34393 requested a full tablet report, sending...
I20260620 01:55:10.689409 11003 heartbeater.cc:507] Master 127.20.1.188:39809 requested a full tablet report, sending...
I20260620 01:55:10.693974 10484 ts_manager.cc:194] Registered new tserver with Master: 0652566484b24e5cb7acc917520c310b (127.20.1.131:35727)
I20260620 01:55:10.694069 10330 ts_manager.cc:194] Registered new tserver with Master: 0652566484b24e5cb7acc917520c310b (127.20.1.131:35727)
I20260620 01:55:10.695751 10330 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:43277
I20260620 01:55:10.698787 11004 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36651
I20260620 01:55:10.699340 11004 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:10.701027 11004 heartbeater.cc:507] Master 127.20.1.189:36651 requested a full tablet report, sending...
I20260620 01:55:10.703647 10405 ts_manager.cc:194] Registered new tserver with Master: 0652566484b24e5cb7acc917520c310b (127.20.1.131:35727)
I20260620 01:55:10.710960 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:55:10.729899 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:55:10.730250 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 10298
I20260620 01:55:10.732707 10360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:55:10.748975 10860 heartbeater.cc:499] Master 127.20.1.190:34393 was elected leader, sending a full tablet report...
I20260620 01:55:10.877895 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 10298
W20260620 01:55:10.899536 10390 connection.cc:573] server connection from 127.0.0.1:45180 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:55:10.901366 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:34393
--webserver_interface=127.20.1.190
--webserver_port=38457
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:55:10.926524 10718 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:34393 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:34393: connect: Connection refused (error 111)
W20260620 01:55:11.399168 11013 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:11.399979 11013 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:11.400209 11013 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:11.411761 11013 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:55:11.411927 11013 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:11.412012 11013 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:55:11.412083 11013 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:55:11.428882 11013 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:34393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=38457
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:11.431030 11013 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:11.433634 11013 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:11.446300 11020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:11.446275 11019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:11.447088 11022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:11.448271 11013 server_base.cc:1061] running on GCE node
I20260620 01:55:11.449273 11013 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:11.449461 11013 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:11.449553 11013 hybrid_clock.cc:648] HybridClock initialized: now 1781920511449532 us; error 0 us; skew 500 ppm
I20260620 01:55:11.449894 11013 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:11 dist-test-slave-wg14 krb5kdc[9999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920511, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:11.455439 11013 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:55:11.459585 11013 webserver.cc:533] Webserver started at http://127.20.1.190:38457/ using document root <none> and password file <none>
I20260620 01:55:11.461194 11013 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:11.461506 11013 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:11.471565 11013 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:55:11.477730 11028 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:11.480582 11013 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260620 01:55:11.480892 11013 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "e939ebcbbd4c4fc3b4da8246c17c2135"
format_stamp: "Formatted at 2026-06-20 01:55:05 on dist-test-slave-wg14"
I20260620 01:55:11.481966 11013 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:11.499895 11013 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:11.511929 11013 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:11.512302 11013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:11.551017 11080 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:34393 every 8 connection(s)
I20260620 01:55:11.551083 11013 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:34393
I20260620 01:55:11.554397 11013 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:55:11.561554 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 11013
I20260620 01:55:11.561995 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 10587
I20260620 01:55:11.567981 11081 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:55:11.575341 11081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Bootstrap starting.
Jun 20 01:55:11 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 01:55:11.600435 11081 log.cc:826] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Log is configured to *not* fsync() on all Append() calls
I20260620 01:55:11.606787 10713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:55:11.616766 11081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:55:11.617954 11081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Bootstrap complete.
I20260620 01:55:11.630595 11081 raft_consensus.cc:359] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:11.632457 11081 raft_consensus.cc:740] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e939ebcbbd4c4fc3b4da8246c17c2135, State: Initialized, Role: FOLLOWER
I20260620 01:55:11.633708 11081 consensus_queue.cc:260] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:11.635068 11087 sys_catalog.cc:455] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:11.635789 11087 sys_catalog.cc:458] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:11.637210 11081 sys_catalog.cc:565] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:55:11.661075 11098 catalog_manager.cc:1294] Loaded cluster ID: a361767a6b444190b4b4e4a25e4ef9c4
I20260620 01:55:11.661556 11098 catalog_manager.cc:1587] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: loading cluster ID for follower catalog manager: success
Jun 20 01:55:11 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920511, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:55:11.683374 11098 catalog_manager.cc:1609] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: acquiring CA information for follower catalog manager: success
I20260620 01:55:11.687351 11098 catalog_manager.cc:1637] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:55:11.703805 11002 heartbeater.cc:499] Master 127.20.1.190:34393 was elected leader, sending a full tablet report...
I20260620 01:55:11.740031 11045 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0652566484b24e5cb7acc917520c310b" instance_seqno: 1781920510608979) as {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:36713; Asking this server to re-register.
I20260620 01:55:11.741498 11002 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:11.744606 11045 ts_manager.cc:194] Registered new tserver with Master: 0652566484b24e5cb7acc917520c310b (127.20.1.131:35727)
I20260620 01:55:11.761270 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 10587
I20260620 01:55:11.789345 11013 master_runner.cc:264] No leader master found from master e939ebcbbd4c4fc3b4da8246c17c2135
I20260620 01:55:11.789525 11013 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:55:11.793057 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35445
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--webserver_port=40645
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:55:11.911331 11045 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f5b050223e074795b659b2c3b4273459" instance_seqno: 1781920509671988) as {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:57381; Asking this server to re-register.
I20260620 01:55:11.912093 11108 raft_consensus.cc:493] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e939ebcbbd4c4fc3b4da8246c17c2135)
I20260620 01:55:11.912361 10860 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:11.912602 11108 raft_consensus.cc:515] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:11.914652 11045 ts_manager.cc:194] Registered new tserver with Master: f5b050223e074795b659b2c3b4273459 (127.20.1.130:41181)
I20260620 01:55:11.918538 10413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a892964204a44e5a8efd106641f0714a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "670ce53faa5041b5aebd1e588d7b1124" is_pre_election: true
I20260620 01:55:11.919152 11108 leader_election.cc:290] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e939ebcbbd4c4fc3b4da8246c17c2135 (127.20.1.190:34393), 670ce53faa5041b5aebd1e588d7b1124 (127.20.1.189:36651)
I20260620 01:55:11.926378 11056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a892964204a44e5a8efd106641f0714a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" is_pre_election: true
I20260620 01:55:11.927081 11056 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a892964204a44e5a8efd106641f0714a in term 1.
I20260620 01:55:11.927886 10470 leader_election.cc:304] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a892964204a44e5a8efd106641f0714a, e939ebcbbd4c4fc3b4da8246c17c2135; no voters: 670ce53faa5041b5aebd1e588d7b1124
I20260620 01:55:11.928686 11108 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:55:11.928812 11108 raft_consensus.cc:493] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 1 FOLLOWER]: Starting leader election (detected failure of leader e939ebcbbd4c4fc3b4da8246c17c2135)
I20260620 01:55:11.928926 11108 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:55:11.931533 11108 raft_consensus.cc:515] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:11.933302 10413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a892964204a44e5a8efd106641f0714a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "670ce53faa5041b5aebd1e588d7b1124"
I20260620 01:55:11.933336 11056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a892964204a44e5a8efd106641f0714a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135"
I20260620 01:55:11.933596 11056 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:55:11.938542 11108 leader_election.cc:290] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [CANDIDATE]: Term 2 election: Requested vote from peers e939ebcbbd4c4fc3b4da8246c17c2135 (127.20.1.190:34393), 670ce53faa5041b5aebd1e588d7b1124 (127.20.1.189:36651)
I20260620 01:55:11.939347 11056 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a892964204a44e5a8efd106641f0714a in term 2.
I20260620 01:55:11.939990 10470 leader_election.cc:304] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a892964204a44e5a8efd106641f0714a, e939ebcbbd4c4fc3b4da8246c17c2135; no voters: 670ce53faa5041b5aebd1e588d7b1124
I20260620 01:55:11.940359 11108 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 01:55:11.940903 11108 raft_consensus.cc:697] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 2 LEADER]: Becoming Leader. State: Replica: a892964204a44e5a8efd106641f0714a, State: Running, Role: LEADER
I20260620 01:55:11.941576 11108 consensus_queue.cc:237] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:11.943800 11110 sys_catalog.cc:455] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a892964204a44e5a8efd106641f0714a. Latest consensus state: current_term: 2 leader_uuid: "a892964204a44e5a8efd106641f0714a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:11.944092 11110 sys_catalog.cc:458] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: This master's current role is: LEADER
I20260620 01:55:11.945080 11112 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:55:11.947458 11112 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:55:11.948561 11112 catalog_manager.cc:1294] Loaded cluster ID: a361767a6b444190b4b4e4a25e4ef9c4
I20260620 01:55:11.948693 11112 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:55:11.950266 11112 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:55:11.951345 11112 catalog_manager.cc:6080] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: Loaded TSK: 0
I20260620 01:55:11.952345 11112 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:55:12.360271 10413 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:55:12.364178 10413 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 2 FOLLOWER]: Refusing update from remote peer a892964204a44e5a8efd106641f0714a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:55:12.365630 11110 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [LEADER]: Connected to new peer: Peer: permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:55:12.369750 11116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a892964204a44e5a8efd106641f0714a. Latest consensus state: current_term: 2 leader_uuid: "a892964204a44e5a8efd106641f0714a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:12.370572 11116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:55:12.371641 11106 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:12.372522 11106 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:12.372828 11106 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260620 01:55:12.373737 11110 sys_catalog.cc:455] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a892964204a44e5a8efd106641f0714a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:12.374107 11110 sys_catalog.cc:458] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: This master's current role is: LEADER
I20260620 01:55:12.376562 11056 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 2 FOLLOWER]: Refusing update from remote peer a892964204a44e5a8efd106641f0714a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:55:12.376521 11116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a892964204a44e5a8efd106641f0714a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:12.377032 11116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:12.377712 11108 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:55:12.383966 11124 sys_catalog.cc:455] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a892964204a44e5a8efd106641f0714a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:12.384891 11124 sys_catalog.cc:458] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:12.384620 11121 sys_catalog.cc:455] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a892964204a44e5a8efd106641f0714a. Latest consensus state: current_term: 2 leader_uuid: "a892964204a44e5a8efd106641f0714a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:12.385267 11121 sys_catalog.cc:458] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:55:12.387090 11106 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:12.387459 11106 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:55:12.400671 11106 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=40645
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:12.403475 11106 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:12.405303 11106 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:12.418193 11127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:12.418288 11126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:12.420068 11106 server_base.cc:1061] running on GCE node
W20260620 01:55:12.420414 11129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:12.421458 11106 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:12.421656 11106 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:12.421991 11106 hybrid_clock.cc:648] HybridClock initialized: now 1781920512421951 us; error 0 us; skew 500 ppm
I20260620 01:55:12.422420 11106 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:12 dist-test-slave-wg14 krb5kdc[9999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920512, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:12.427345 11106 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:55:12.429808 11106 webserver.cc:533] Webserver started at http://127.20.1.129:40645/ using document root <none> and password file <none>
I20260620 01:55:12.430838 11106 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:12.431018 11106 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:12.438920 11106 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:55:12.444620 11135 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:12.447187 11106 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260620 01:55:12.447422 11106 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "39486b58a61649dfababf1d4b5322a97"
format_stamp: "Formatted at 2026-06-20 01:55:08 on dist-test-slave-wg14"
I20260620 01:55:12.448344 11106 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:12.476491 11106 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:12.483002 11106 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:12.483415 11106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:12.485170 11106 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:55:12.487913 11106 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:55:12.488082 11106 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:12.488200 11106 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:55:12.488260 11106 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:12.535207 11106 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35445
I20260620 01:55:12.535516 11248 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35445 every 8 connection(s)
I20260620 01:55:12.537169 11106 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:55:12.542130 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 11106
Jun 20 01:55:12 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Jun 20 01:55:12 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920512, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:55:12 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920512, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:55:12 dist-test-slave-wg14 krb5kdc[9999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920512, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:55:12.608007 11251 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36651
I20260620 01:55:12.608060 11250 heartbeater.cc:344] Connected to a master server at 127.20.1.188:39809
I20260620 01:55:12.608561 11251 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:12.608572 11250 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:12.610046 11251 heartbeater.cc:507] Master 127.20.1.189:36651 requested a full tablet report, sending...
I20260620 01:55:12.610307 11250 heartbeater.cc:507] Master 127.20.1.188:39809 requested a full tablet report, sending...
I20260620 01:55:12.612226 11249 heartbeater.cc:344] Connected to a master server at 127.20.1.190:34393
I20260620 01:55:12.612486 11249 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:12.613212 11249 heartbeater.cc:507] Master 127.20.1.190:34393 requested a full tablet report, sending...
I20260620 01:55:12.613940 10484 ts_manager.cc:194] Re-registered known tserver with Master: 39486b58a61649dfababf1d4b5322a97 (127.20.1.129:35445)
I20260620 01:55:12.614768 10405 ts_manager.cc:194] Re-registered known tserver with Master: 39486b58a61649dfababf1d4b5322a97 (127.20.1.129:35445)
I20260620 01:55:12.616324 11045 ts_manager.cc:194] Registered new tserver with Master: 39486b58a61649dfababf1d4b5322a97 (127.20.1.129:35445)
I20260620 01:55:12.616672 10484 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:41767
I20260620 01:55:12.759738 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 11106
W20260620 01:55:12.762518 11263 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:55:12.756561 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:48714 (local address 127.20.1.129:35445)
0620 01:55:12.757387 (+ 826us) server_negotiation.cc:207] Beginning negotiation
0620 01:55:12.757395 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:55:12.757428 (+ 33us) server_negotiation.cc:408] Connection header received
0620 01:55:12.757506 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:55:12.757514 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:55:12.757790 (+ 276us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:55:12.758055 (+ 265us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:55:12.759571 (+ 1516us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48714: BlockingRecv error: recv got EOF from 127.0.0.1:48714 (error 108)
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":401,"threads_started":1}
W20260620 01:55:12.763860 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 11106: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:35445: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:55:12.764070 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 11106
I20260620 01:55:12.781670 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35445
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--webserver_port=40645
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:55:12.838284 11119 sys_catalog.cc:455] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a892964204a44e5a8efd106641f0714a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:12.838798 11119 sys_catalog.cc:458] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: This master's current role is: LEADER
W20260620 01:55:13.243809 11265 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:13.244444 11265 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:13.244707 11265 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:55:13.244917 11265 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:13.255699 11265 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:13.256004 11265 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:55:13.270669 11265 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=40645
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:13.272826 11265 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:13.274576 11265 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:13.288305 11273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:13.288359 11271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:13.288401 11270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:13.290422 11265 server_base.cc:1061] running on GCE node
I20260620 01:55:13.291574 11265 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:13.291885 11265 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:13.292336 11265 hybrid_clock.cc:648] HybridClock initialized: now 1781920513292094 us; error 0 us; skew 500 ppm
I20260620 01:55:13.292841 11265 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:13 dist-test-slave-wg14 krb5kdc[9999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920513, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:13.297631 11265 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:55:13.300124 11265 webserver.cc:533] Webserver started at http://127.20.1.129:40645/ using document root <none> and password file <none>
I20260620 01:55:13.301334 11265 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:13.301532 11265 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:13.309337 11265 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 01:55:13.314548 11279 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:13.316743 11265 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:55:13.317041 11265 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "39486b58a61649dfababf1d4b5322a97"
format_stamp: "Formatted at 2026-06-20 01:55:08 on dist-test-slave-wg14"
I20260620 01:55:13.318208 11265 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:13.339720 11265 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:13.345880 11265 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:13.346297 11265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:13.347967 11265 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:55:13.351369 11265 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:55:13.351539 11265 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:13.351682 11265 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:55:13.351758 11265 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:55:13.352330 11265 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:55:13.352413 11265 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:55:13.352486 11265 fault_injection.cc:59] injecting fault for kudu (pid 11265): FLAGS_fault_before_start (process will exit)
I20260620 01:55:13.373903 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35445
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--webserver_port=40645
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:55:13.703554 11003 heartbeater.cc:499] Master 127.20.1.188:39809 was elected leader, sending a full tablet report...
I20260620 01:55:13.751096 10861 heartbeater.cc:499] Master 127.20.1.188:39809 was elected leader, sending a full tablet report...
W20260620 01:55:13.820891 11287 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:13.821322 11287 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:13.821477 11287 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:13.831951 11287 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:13.832217 11287 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:55:13.846410 11287 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=40645
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:13.848615 11287 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:13.850328 11287 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:13.863859 11293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:13.863806 11295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:13.863888 11292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:13.865613 11287 server_base.cc:1061] running on GCE node
I20260620 01:55:13.866511 11287 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:13.866720 11287 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:13.866822 11287 hybrid_clock.cc:648] HybridClock initialized: now 1781920513866797 us; error 0 us; skew 500 ppm
I20260620 01:55:13.867244 11287 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:13 dist-test-slave-wg14 krb5kdc[9999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920513, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:13.872361 11287 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:55:13.875087 11287 webserver.cc:533] Webserver started at http://127.20.1.129:40645/ using document root <none> and password file <none>
I20260620 01:55:13.876468 11287 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:13.876736 11287 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:13.884490 11287 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:55:13.889420 11301 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:13.892109 11287 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260620 01:55:13.892405 11287 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "39486b58a61649dfababf1d4b5322a97"
format_stamp: "Formatted at 2026-06-20 01:55:08 on dist-test-slave-wg14"
I20260620 01:55:13.893440 11287 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:13.914533 11287 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:13.921129 11287 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:13.921478 11287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:13.923063 11287 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:55:13.925972 11287 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:55:13.926141 11287 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:13.926244 11287 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:55:13.926301 11287 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:13.973788 11287 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35445
I20260620 01:55:13.973934 11414 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35445 every 8 connection(s)
I20260620 01:55:13.976541 11287 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:55:13.979727 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 11287
I20260620 01:55:13.980283 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 11287
W20260620 01:55:14.002859 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 11287: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:35445: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:55:14.003042 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 11287
W20260620 01:55:14.019759 11425 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:55:13.992695 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:33567 (local address 127.20.1.189:36651)
0620 01:55:14.000397 (+ 7702us) server_negotiation.cc:207] Beginning negotiation
0620 01:55:14.000405 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:55:14.018622 (+ 18217us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:33567: BlockingRecv error: recv got EOF from 127.20.1.129:33567 (error 108)
Metrics: {"server-negotiator.queue_time_us":7534,"thread_start_us":7383,"threads_started":1}
I20260620 01:55:14.019899 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 10732
W20260620 01:55:14.020219 11421 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:55:13.983869 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:44391 (local address 127.20.1.188:39809)
0620 01:55:13.987209 (+ 3340us) server_negotiation.cc:207] Beginning negotiation
0620 01:55:13.987220 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0620 01:55:14.001316 (+ 14096us) server_negotiation.cc:408] Connection header received
0620 01:55:14.001730 (+ 414us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:55:14.001739 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:55:14.001841 (+ 102us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:55:14.001972 (+ 131us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:55:14.018921 (+ 16949us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:44391: BlockingRecv error: recv got EOF from 127.20.1.129:44391 (error 108)
Metrics: {"server-negotiator.queue_time_us":3079,"thread_start_us":2961,"threads_started":1}
W20260620 01:55:14.020267 11420 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:55:13.983333 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:43653 (local address 127.20.1.190:34393)
0620 01:55:13.997073 (+ 13740us) server_negotiation.cc:207] Beginning negotiation
0620 01:55:13.997081 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:55:13.997125 (+ 44us) server_negotiation.cc:408] Connection header received
0620 01:55:14.001902 (+ 4777us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:55:14.001909 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:55:14.002040 (+ 131us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:55:14.002167 (+ 127us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:55:14.018958 (+ 16791us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:43653: BlockingRecv error: recv got EOF from 127.20.1.129:43653 (error 108)
Metrics: {"server-negotiator.queue_time_us":13502,"thread_start_us":13809,"threads_started":1}
W20260620 01:55:14.024119 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 10732: Not authorized: Client connection negotiation failed: client connection to 127.20.1.130:41181: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:55:14.024421 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 10732
I20260620 01:55:14.043963 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 10874
W20260620 01:55:14.047993 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 10874: Not authorized: Client connection negotiation failed: client connection to 127.20.1.131:35727: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:55:14.048202 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 10874
I20260620 01:55:14.063464 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 11013
W20260620 01:55:14.066444 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 11013: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:34393: server requires authentication, but client does not have Kerberos credentials available
W20260620 01:55:14.066567 11420 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:55:14.064789 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:57656 (local address 127.20.1.190:34393)
0620 01:55:14.064993 (+ 204us) server_negotiation.cc:207] Beginning negotiation
0620 01:55:14.065000 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 01:55:14.065027 (+ 27us) server_negotiation.cc:408] Connection header received
0620 01:55:14.065180 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:55:14.065185 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:55:14.065265 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:55:14.065361 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:55:14.066304 (+ 943us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57656: BlockingRecv error: recv got EOF from 127.0.0.1:57656 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 01:55:14.066658 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 11013
I20260620 01:55:14.089066 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 10374
I20260620 01:55:14.090719 10436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:55:14.229290 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 10374
I20260620 01:55:14.250710 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 10451
I20260620 01:55:14.253005 10515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:55:14.408854 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 10451
Shutting down hive metastore.
Jun 20 01:55:14 dist-test-slave-wg14 krb5kdc[11436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 01:55:14 dist-test-slave-wg14 krb5kdc[11436](info): set up 2 sockets
Jun 20 01:55:14 dist-test-slave-wg14 krb5kdc[11436](info): commencing operation
krb5kdc: starting...
W20260620 01:55:16.887678 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.060s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 01:55:17 dist-test-slave-wg14 krb5kdc[11436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "hive/127.0.0.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf
2026-06-20 01:55:18: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Jun 20 01:55:21 dist-test-slave-wg14 krb5kdc[11436](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920521, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 01:55:26 dist-test-slave-wg14 krb5kdc[11436](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920526, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 01:55:26 dist-test-slave-wg14 krb5kdc[11436](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920526, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.190@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.190@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:55:28.553174 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:34393
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:55:28.983814 11632 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:28.984474 11632 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:28.984879 11632 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:28.995586 11632 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:55:28.995759 11632 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:28.995954 11632 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:55:28.996114 11632 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:55:29.010281 11632 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:34393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:29.012885 11632 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:29.015517 11632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:29.028733 11637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:29.028728 11638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:29.030377 11640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:29.032456 11632 server_base.cc:1061] running on GCE node
I20260620 01:55:29.033514 11632 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:29.033722 11632 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:29.033821 11632 hybrid_clock.cc:648] HybridClock initialized: now 1781920529033794 us; error 0 us; skew 500 ppm
I20260620 01:55:29.034225 11632 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:29 dist-test-slave-wg14 krb5kdc[11436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920529, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:29.038594 11632 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:55:29.041180 11632 webserver.cc:533] Webserver started at http://127.20.1.190:42805/ using document root <none> and password file <none>
I20260620 01:55:29.042124 11632 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:29.042270 11632 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:29.049863 11632 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 01:55:29.054654 11646 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:29.057178 11632 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260620 01:55:29.057510 11632 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "e939ebcbbd4c4fc3b4da8246c17c2135"
format_stamp: "Formatted at 2026-06-20 01:55:05 on dist-test-slave-wg14"
I20260620 01:55:29.058740 11632 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:29.090972 11632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:29.100418 11632 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:29.100853 11632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:29.128053 11632 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:34393
I20260620 01:55:29.128185 11698 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:34393 every 8 connection(s)
I20260620 01:55:29.130153 11632 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:55:29.138729 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 11632
I20260620 01:55:29.138981 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:55:29.140565 11699 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:55:29.145984 11699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Bootstrap starting.
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.189@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:55:29.169569 11699 log.cc:826] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Log is configured to *not* fsync() on all Append() calls
I20260620 01:55:29.184510 11699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:55:29.187060 11699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Bootstrap complete.
I20260620 01:55:29.198659 11699 raft_consensus.cc:359] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:29.200784 11699 raft_consensus.cc:740] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e939ebcbbd4c4fc3b4da8246c17c2135, State: Initialized, Role: FOLLOWER
I20260620 01:55:29.202263 11699 consensus_queue.cc:260] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:29.205200 11704 sys_catalog.cc:455] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:29.205708 11704 sys_catalog.cc:458] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:29.207901 11699 sys_catalog.cc:565] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: configured and running, proceeding with master startup.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:55:29.231974 11715 catalog_manager.cc:1294] Loaded cluster ID: a361767a6b444190b4b4e4a25e4ef9c4
I20260620 01:55:29.232301 11715 catalog_manager.cc:1587] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: loading cluster ID for follower catalog manager: success
I20260620 01:55:29.238225 11715 catalog_manager.cc:1609] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: acquiring CA information for follower catalog manager: success
I20260620 01:55:29.243389 11715 catalog_manager.cc:1637] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Jun 20 01:55:29 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920529, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.189@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:55:29.271234 11632 master_runner.cc:187] Error getting master registration for 127.20.1.189:36651: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36651: connect: Connection refused (error 111),
I20260620 01:55:29.277456 11632 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:55:29.302969 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:36651
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:55:29.781818 11721 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:29.782356 11721 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:29.782925 11721 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:29.793504 11721 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:55:29.793916 11721 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:29.794224 11721 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:55:29.794344 11721 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:55:29.809516 11721 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:36651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:29.812482 11721 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:29.814631 11721 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:29.829007 11727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:29.829231 11726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:29.831950 11729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:29.831940 11721 server_base.cc:1061] running on GCE node
I20260620 01:55:29.834617 11721 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:29.835186 11721 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:29.835640 11721 hybrid_clock.cc:648] HybridClock initialized: now 1781920529835597 us; error 0 us; skew 500 ppm
I20260620 01:55:29.836668 11721 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:29 dist-test-slave-wg14 krb5kdc[11436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920529, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:29.842362 11721 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 01:55:29.846364 11721 webserver.cc:533] Webserver started at http://127.20.1.189:42645/ using document root <none> and password file <none>
I20260620 01:55:29.847898 11721 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:29.848093 11721 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:29.857131 11721 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.002s sys 0.005s
I20260620 01:55:29.862061 11735 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:29.864712 11721 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:55:29.864993 11721 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "670ce53faa5041b5aebd1e588d7b1124"
format_stamp: "Formatted at 2026-06-20 01:55:06 on dist-test-slave-wg14"
I20260620 01:55:29.866340 11721 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:29.902675 11721 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:29.909933 11721 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:29.910241 11721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:29.941823 11721 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:36651
I20260620 01:55:29.941841 11787 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:36651 every 8 connection(s)
I20260620 01:55:29.943877 11721 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:55:29.948791 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 11721
I20260620 01:55:29.949023 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:55:29.955504 11788 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:55:29.961094 11788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124: Bootstrap starting.
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.188@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:55:29.985211 11788 log.cc:826] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124: Log is configured to *not* fsync() on all Append() calls
I20260620 01:55:30.001031 11788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:55:30.001956 11788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124: Bootstrap complete.
I20260620 01:55:30.013904 11788 raft_consensus.cc:359] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:30.016120 11788 raft_consensus.cc:740] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 670ce53faa5041b5aebd1e588d7b1124, State: Initialized, Role: FOLLOWER
I20260620 01:55:30.017333 11788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:30.018407 11793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:30.019167 11793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:30.020061 11788 sys_catalog.cc:565] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: configured and running, proceeding with master startup.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:55:30.053004 11804 catalog_manager.cc:1294] Loaded cluster ID: a361767a6b444190b4b4e4a25e4ef9c4
I20260620 01:55:30.053279 11804 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124: loading cluster ID for follower catalog manager: success
I20260620 01:55:30.059813 11804 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124: acquiring CA information for follower catalog manager: success
Jun 20 01:55:30 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920529, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:55:30.064359 11804 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.188@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:55:30.107367 11721 master_runner.cc:187] Error getting master registration for 127.20.1.188:39809: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39809: connect: Connection refused (error 111),
I20260620 01:55:30.112501 11721 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:55:30.115767 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:39809
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:55:30.556986 11815 raft_consensus.cc:493] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:55:30.557412 11815 raft_consensus.cc:515] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:30.561108 11815 leader_election.cc:290] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 670ce53faa5041b5aebd1e588d7b1124 (127.20.1.189:36651), a892964204a44e5a8efd106641f0714a (127.20.1.188:39809)
W20260620 01:55:30.560940 11811 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:30.561510 11811 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:30.561754 11811 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:30.572741 11647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39809: connect: Connection refused (error 111)
W20260620 01:55:30.579458 11811 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:55:30.579947 11811 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:30.580178 11811 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:55:30.580476 11811 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260620 01:55:30.584229 11647 leader_election.cc:336] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a892964204a44e5a8efd106641f0714a (127.20.1.188:39809): Network error: Client connection negotiation failed: client connection to 127.20.1.188:39809: connect: Connection refused (error 111)
I20260620 01:55:30.586464 11762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "670ce53faa5041b5aebd1e588d7b1124" is_pre_election: true
I20260620 01:55:30.587307 11762 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e939ebcbbd4c4fc3b4da8246c17c2135 in term 2.
I20260620 01:55:30.588810 11649 leader_election.cc:304] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 670ce53faa5041b5aebd1e588d7b1124, e939ebcbbd4c4fc3b4da8246c17c2135; no voters: a892964204a44e5a8efd106641f0714a
I20260620 01:55:30.589545 11815 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 01:55:30.589797 11815 raft_consensus.cc:493] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:55:30.589936 11815 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:55:30.594692 11815 raft_consensus.cc:515] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:30.595659 11815 leader_election.cc:290] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [CANDIDATE]: Term 3 election: Requested vote from peers 670ce53faa5041b5aebd1e588d7b1124 (127.20.1.189:36651), a892964204a44e5a8efd106641f0714a (127.20.1.188:39809)
I20260620 01:55:30.596732 11762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "670ce53faa5041b5aebd1e588d7b1124"
I20260620 01:55:30.597054 11762 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 2 FOLLOWER]: Advancing to term 3
W20260620 01:55:30.598997 11647 leader_election.cc:336] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a892964204a44e5a8efd106641f0714a (127.20.1.188:39809): Network error: Client connection negotiation failed: client connection to 127.20.1.188:39809: connect: Connection refused (error 111)
I20260620 01:55:30.600827 11811 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:39809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:30.603067 11762 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e939ebcbbd4c4fc3b4da8246c17c2135 in term 3.
I20260620 01:55:30.603200 11811 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:30.603945 11649 leader_election.cc:304] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 670ce53faa5041b5aebd1e588d7b1124, e939ebcbbd4c4fc3b4da8246c17c2135; no voters: a892964204a44e5a8efd106641f0714a
I20260620 01:55:30.604717 11815 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 01:55:30.606043 11811 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260620 01:55:30.607582 11815 raft_consensus.cc:697] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 3 LEADER]: Becoming Leader. State: Replica: e939ebcbbd4c4fc3b4da8246c17c2135, State: Running, Role: LEADER
I20260620 01:55:30.608343 11815 consensus_queue.cc:237] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:30.612911 11819 sys_catalog.cc:455] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e939ebcbbd4c4fc3b4da8246c17c2135. Latest consensus state: current_term: 3 leader_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:30.613613 11819 sys_catalog.cc:458] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: This master's current role is: LEADER
I20260620 01:55:30.614656 11821 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:55:30.618383 11821 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:55:30.619841 11821 catalog_manager.cc:1294] Loaded cluster ID: a361767a6b444190b4b4e4a25e4ef9c4
I20260620 01:55:30.619967 11821 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:55:30.621572 11821 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:55:30.623198 11821 catalog_manager.cc:6080] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Loaded TSK: 0
I20260620 01:55:30.625671 11821 catalog_manager.cc:1549] Initializing in-progress tserver states...
W20260620 01:55:30.634979 11823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:30.635641 11824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:30.636650 11826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:30.643870 11811 server_base.cc:1061] running on GCE node
I20260620 01:55:30.646116 11811 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:30.646425 11811 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:30.646540 11811 hybrid_clock.cc:648] HybridClock initialized: now 1781920530646515 us; error 0 us; skew 500 ppm
I20260620 01:55:30.647611 11811 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:30 dist-test-slave-wg14 krb5kdc[11436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920530, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:30.653198 11811 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 01:55:30.656396 11811 webserver.cc:533] Webserver started at http://127.20.1.188:33785/ using document root <none> and password file <none>
I20260620 01:55:30.657401 11811 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:30.657577 11811 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:30.667351 11811 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 01:55:30.671772 11832 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:30.673939 11811 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260620 01:55:30.674237 11811 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "a892964204a44e5a8efd106641f0714a"
format_stamp: "Formatted at 2026-06-20 01:55:07 on dist-test-slave-wg14"
I20260620 01:55:30.675419 11811 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:30.695019 11811 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:30.701624 11811 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:30.702178 11811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:30.729800 11811 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:39809
I20260620 01:55:30.729787 11884 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:39809 every 8 connection(s)
I20260620 01:55:30.731990 11811 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:55:30.736424 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 11811
I20260620 01:55:30.736661 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:55:30.742484 11885 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:55:30.747673 11885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: Bootstrap starting.
Jun 20 01:55:30 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:55:30.769429 11885 log.cc:826] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: Log is configured to *not* fsync() on all Append() calls
Jun 20 01:55:30 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:55:30.782758 11885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:55:30.783588 11885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: Bootstrap complete.
I20260620 01:55:30.796149 11885 raft_consensus.cc:359] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:30.797868 11885 raft_consensus.cc:740] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a892964204a44e5a8efd106641f0714a, State: Initialized, Role: FOLLOWER
I20260620 01:55:30.799054 11885 consensus_queue.cc:260] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.129@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:55:30.800403 11892 sys_catalog.cc:455] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:30.800782 11892 sys_catalog.cc:458] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:30.801868 11885 sys_catalog.cc:565] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 01:55:30 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:55:30.840158 11903 catalog_manager.cc:1294] Loaded cluster ID: a361767a6b444190b4b4e4a25e4ef9c4
I20260620 01:55:30.840273 11903 catalog_manager.cc:1587] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: loading cluster ID for follower catalog manager: success
I20260620 01:55:30.846432 11903 catalog_manager.cc:1609] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: acquiring CA information for follower catalog manager: success
Jun 20 01:55:30 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920530, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:55:30.850911 11903 catalog_manager.cc:1637] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.129@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:55:30.929873 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:55:31.111415 11860 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:55:31.120256 11860 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 3 FOLLOWER]: Refusing update from remote peer e939ebcbbd4c4fc3b4da8246c17c2135: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:55:31.122066 11819 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:55:31.127400 11892 sys_catalog.cc:455] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e939ebcbbd4c4fc3b4da8246c17c2135. Latest consensus state: current_term: 3 leader_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:31.127861 11892 sys_catalog.cc:458] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:31.154625 11892 sys_catalog.cc:455] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:31.155009 11892 sys_catalog.cc:458] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:31.160879 11762 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 3 FOLLOWER]: Refusing update from remote peer e939ebcbbd4c4fc3b4da8246c17c2135: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:55:31.162371 11819 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:55:31.165055 11911 sys_catalog.cc:455] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:31.165589 11911 sys_catalog.cc:458] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: This master's current role is: LEADER
I20260620 01:55:31.172925 11919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:31.173688 11919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:31.174173 11915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e939ebcbbd4c4fc3b4da8246c17c2135. Latest consensus state: current_term: 3 leader_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:31.174518 11915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:55:31.469030 11909 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:31.469434 11909 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:31.469555 11909 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:31.479972 11909 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:31.480221 11909 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:55:31.493257 11909 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:31.495479 11909 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:31.497102 11909 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:31.511770 11924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:31.511801 11925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:31.511981 11927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:31.512933 11909 server_base.cc:1061] running on GCE node
I20260620 01:55:31.513808 11909 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:31.514004 11909 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:31.514138 11909 hybrid_clock.cc:648] HybridClock initialized: now 1781920531514104 us; error 0 us; skew 500 ppm
I20260620 01:55:31.514621 11909 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:31 dist-test-slave-wg14 krb5kdc[11436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920531, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:31.520704 11909 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:55:31.524250 11909 webserver.cc:533] Webserver started at http://127.20.1.129:46275/ using document root <none> and password file <none>
I20260620 01:55:31.525247 11909 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:31.525399 11909 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:31.533806 11909 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 01:55:31.539463 11933 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:31.541296 11909 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260620 01:55:31.541507 11909 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "39486b58a61649dfababf1d4b5322a97"
format_stamp: "Formatted at 2026-06-20 01:55:08 on dist-test-slave-wg14"
I20260620 01:55:31.542394 11909 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:31.591413 11909 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:31.598587 11909 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:31.598948 11909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:31.600345 11909 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:55:31.602950 11909 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:55:31.603088 11909 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:31.603204 11909 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:55:31.603354 11909 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:31.661774 11909 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38095
I20260620 01:55:31.661924 12049 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38095 every 8 connection(s)
I20260620 01:55:31.663938 11909 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:55:31.673723 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 11909
I20260620 01:55:31.674077 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 01:55:31 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920531, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:55:31 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920531, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:55:31.717897 12051 heartbeater.cc:344] Connected to a master server at 127.20.1.188:39809
I20260620 01:55:31.718966 12051 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:31.720675 12051 heartbeater.cc:507] Master 127.20.1.188:39809 requested a full tablet report, sending...
Jun 20 01:55:31 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920531, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:55:31.722570 12052 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36651
I20260620 01:55:31.722872 12052 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:31.723690 12052 heartbeater.cc:507] Master 127.20.1.189:36651 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.1.130@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.130@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:55:31.727008 11752 ts_manager.cc:194] Registered new tserver with Master: 39486b58a61649dfababf1d4b5322a97 (127.20.1.129:38095)
I20260620 01:55:31.727758 11849 ts_manager.cc:194] Registered new tserver with Master: 39486b58a61649dfababf1d4b5322a97 (127.20.1.129:38095)
I20260620 01:55:31.724186 12061 sys_catalog.cc:455] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:31.735801 12061 sys_catalog.cc:458] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: This master's current role is: LEADER
I20260620 01:55:31.742259 12050 heartbeater.cc:344] Connected to a master server at 127.20.1.190:34393
I20260620 01:55:31.742453 12050 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:31.742939 12050 heartbeater.cc:507] Master 127.20.1.190:34393 requested a full tablet report, sending...
I20260620 01:55:31.747303 11663 ts_manager.cc:194] Registered new tserver with Master: 39486b58a61649dfababf1d4b5322a97 (127.20.1.129:38095)
I20260620 01:55:31.751005 11663 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:35741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.130 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.130 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.130@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.130@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.130 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.130 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:55:31.841590 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:55:32.326958 12069 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:32.327448 12069 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:32.327571 12069 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:32.337986 12069 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:32.338264 12069 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:55:32.352231 12069 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:32.354341 12069 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:32.356366 12069 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:32.374972 12075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:32.375470 12074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:32.382012 12077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:32.383426 12069 server_base.cc:1061] running on GCE node
I20260620 01:55:32.384183 12069 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:32.384373 12069 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:32.384475 12069 hybrid_clock.cc:648] HybridClock initialized: now 1781920532384453 us; error 0 us; skew 500 ppm
I20260620 01:55:32.384861 12069 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:32 dist-test-slave-wg14 krb5kdc[11436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920532, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:32.391055 12069 init.cc:377] Logged in from keytab as kudu/127.20.1.130@KRBTEST.COM (short username kudu)
I20260620 01:55:32.394393 12069 webserver.cc:533] Webserver started at http://127.20.1.130:40303/ using document root <none> and password file <none>
I20260620 01:55:32.395598 12069 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:32.395762 12069 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:32.404014 12069 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:55:32.409175 12083 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:32.412088 12069 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:55:32.412389 12069 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "f5b050223e074795b659b2c3b4273459"
format_stamp: "Formatted at 2026-06-20 01:55:09 on dist-test-slave-wg14"
I20260620 01:55:32.413743 12069 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:32.443374 12069 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:32.450856 12069 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:32.451448 12069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:32.453365 12069 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:55:32.456230 12069 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:55:32.456390 12069 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:32.456519 12069 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:55:32.456590 12069 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:32.514228 12069 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:46165
I20260620 01:55:32.514420 12196 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:46165 every 8 connection(s)
I20260620 01:55:32.516424 12069 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:55:32.522163 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 12069
I20260620 01:55:32.522620 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
Jun 20 01:55:32 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920532, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:55:32 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920532, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:55:32 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920532, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:55:32.566988 12197 heartbeater.cc:344] Connected to a master server at 127.20.1.190:34393
I20260620 01:55:32.567590 12197 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:32.569913 12197 heartbeater.cc:507] Master 127.20.1.190:34393 requested a full tablet report, sending...
I20260620 01:55:32.571458 12199 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36651
I20260620 01:55:32.571677 12199 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:32.572398 12199 heartbeater.cc:507] Master 127.20.1.189:36651 requested a full tablet report, sending...
I20260620 01:55:32.572904 11663 ts_manager.cc:194] Registered new tserver with Master: f5b050223e074795b659b2c3b4273459 (127.20.1.130:46165)
I20260620 01:55:32.574050 11752 ts_manager.cc:194] Registered new tserver with Master: f5b050223e074795b659b2c3b4273459 (127.20.1.130:46165)
I20260620 01:55:32.574179 11663 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:60415
I20260620 01:55:32.574666 12198 heartbeater.cc:344] Connected to a master server at 127.20.1.188:39809
I20260620 01:55:32.575099 12198 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:32.576229 12198 heartbeater.cc:507] Master 127.20.1.188:39809 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.1.131@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.131@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:55:32.577968 11849 ts_manager.cc:194] Registered new tserver with Master: f5b050223e074795b659b2c3b4273459 (127.20.1.130:46165)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.131 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.131 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.131@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.131@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.131 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.131 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:55:32.695634 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:55:32.755007 12050 heartbeater.cc:499] Master 127.20.1.190:34393 was elected leader, sending a full tablet report...
W20260620 01:55:33.145772 12211 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:33.146144 12211 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:33.146296 12211 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:33.156356 12211 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:33.156723 12211 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:55:33.170384 12211 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:33.172700 12211 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:33.174630 12211 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:33.190271 12217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:33.191658 12216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:33.193185 12219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:33.195808 12211 server_base.cc:1061] running on GCE node
I20260620 01:55:33.196832 12211 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:33.197053 12211 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:33.197171 12211 hybrid_clock.cc:648] HybridClock initialized: now 1781920533197146 us; error 0 us; skew 500 ppm
I20260620 01:55:33.197523 12211 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:33 dist-test-slave-wg14 krb5kdc[11436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920533, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:33.203779 12211 init.cc:377] Logged in from keytab as kudu/127.20.1.131@KRBTEST.COM (short username kudu)
I20260620 01:55:33.210453 12211 webserver.cc:533] Webserver started at http://127.20.1.131:42567/ using document root <none> and password file <none>
I20260620 01:55:33.211737 12211 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:33.211941 12211 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:33.220117 12211 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 01:55:33.225373 12225 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:33.228677 12211 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.002s sys 0.004s
I20260620 01:55:33.228964 12211 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "0652566484b24e5cb7acc917520c310b"
format_stamp: "Formatted at 2026-06-20 01:55:10 on dist-test-slave-wg14"
I20260620 01:55:33.229899 12211 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:33.267637 12211 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:33.272660 12211 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:33.273005 12211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:33.274798 12211 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:55:33.277750 12211 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:55:33.277932 12211 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:33.278072 12211 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:55:33.278152 12211 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:33.333793 12211 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:35939
I20260620 01:55:33.333895 12339 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:35939 every 8 connection(s)
I20260620 01:55:33.335760 12211 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:55:33.342614 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 12211
I20260620 01:55:33.343003 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
Jun 20 01:55:33 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920533, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:55:33 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920533, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:55:33 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920533, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:55:33.382213 12340 heartbeater.cc:344] Connected to a master server at 127.20.1.190:34393
I20260620 01:55:33.382272 12341 heartbeater.cc:344] Connected to a master server at 127.20.1.188:39809
I20260620 01:55:33.382642 12340 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:33.382702 12341 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:33.384220 12340 heartbeater.cc:507] Master 127.20.1.190:34393 requested a full tablet report, sending...
I20260620 01:55:33.384228 12341 heartbeater.cc:507] Master 127.20.1.188:39809 requested a full tablet report, sending...
I20260620 01:55:33.386999 11663 ts_manager.cc:194] Registered new tserver with Master: 0652566484b24e5cb7acc917520c310b (127.20.1.131:35939)
I20260620 01:55:33.387259 11848 ts_manager.cc:194] Registered new tserver with Master: 0652566484b24e5cb7acc917520c310b (127.20.1.131:35939)
I20260620 01:55:33.388706 11663 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:48461
I20260620 01:55:33.390290 12342 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36651
I20260620 01:55:33.390560 12342 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:33.392756 12342 heartbeater.cc:507] Master 127.20.1.189:36651 requested a full tablet report, sending...
I20260620 01:55:33.394367 11752 ts_manager.cc:194] Registered new tserver with Master: 0652566484b24e5cb7acc917520c310b (127.20.1.131:35939)
I20260620 01:55:33.414753 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:55:33.428120 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:55:33.428331 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 11632
I20260620 01:55:33.429557 11694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:55:33.561475 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 11632
I20260620 01:55:33.581041 12197 heartbeater.cc:499] Master 127.20.1.190:34393 was elected leader, sending a full tablet report...
W20260620 01:55:33.582309 12086 connection.cc:573] client connection to 127.20.1.190:34393 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 01:55:33.583375 12197 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:34393 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:55:33.583971 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:34393
--webserver_interface=127.20.1.190
--webserver_port=42805
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:55:33.760705 12050 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:34393 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:34393: connect: Connection refused (error 111)
W20260620 01:55:34.072698 12351 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:34.073128 12351 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:34.073264 12351 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:34.083642 12351 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:55:34.083822 12351 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:34.083932 12351 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:55:34.083997 12351 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:55:34.098440 12351 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:34393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=42805
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:34.100425 12351 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:34.102406 12351 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:34.112538 12358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:34.112538 12357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:34.114218 12360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:34.114776 12351 server_base.cc:1061] running on GCE node
I20260620 01:55:34.115767 12351 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:34.115952 12351 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:34.116042 12351 hybrid_clock.cc:648] HybridClock initialized: now 1781920534116022 us; error 0 us; skew 500 ppm
I20260620 01:55:34.116410 12351 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:34 dist-test-slave-wg14 krb5kdc[11436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920534, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:34.121812 12351 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:55:34.126173 12351 webserver.cc:533] Webserver started at http://127.20.1.190:42805/ using document root <none> and password file <none>
I20260620 01:55:34.127435 12351 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:34.127650 12351 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:34.135267 12351 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 01:55:34.140093 12366 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:34.142707 12351 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260620 01:55:34.142999 12351 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "e939ebcbbd4c4fc3b4da8246c17c2135"
format_stamp: "Formatted at 2026-06-20 01:55:05 on dist-test-slave-wg14"
I20260620 01:55:34.143951 12351 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:34.166697 12351 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:34.177047 12351 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:34.177395 12351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:34.209852 12351 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:34393
I20260620 01:55:34.209893 12418 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:34393 every 8 connection(s)
I20260620 01:55:34.212344 12351 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:55:34.218528 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 12351
I20260620 01:55:34.218935 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 11909
I20260620 01:55:34.222597 12419 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:55:34.233419 12419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Bootstrap starting.
Jun 20 01:55:34 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 01:55:34.251904 12044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:55:34.259925 12419 log.cc:826] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Log is configured to *not* fsync() on all Append() calls
I20260620 01:55:34.275971 12419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:55:34.276857 12419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: Bootstrap complete.
I20260620 01:55:34.286239 12419 raft_consensus.cc:359] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:34.287245 12419 raft_consensus.cc:740] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e939ebcbbd4c4fc3b4da8246c17c2135, State: Initialized, Role: FOLLOWER
I20260620 01:55:34.288350 12419 consensus_queue.cc:260] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:34.289439 12425 sys_catalog.cc:455] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:34.289821 12425 sys_catalog.cc:458] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:34.290798 12419 sys_catalog.cc:565] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:55:34.308027 12436 catalog_manager.cc:1294] Loaded cluster ID: a361767a6b444190b4b4e4a25e4ef9c4
I20260620 01:55:34.308398 12436 catalog_manager.cc:1587] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: loading cluster ID for follower catalog manager: success
I20260620 01:55:34.316217 12436 catalog_manager.cc:1609] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: acquiring CA information for follower catalog manager: success
I20260620 01:55:34.319824 12436 catalog_manager.cc:1637] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:55:34.383191 12351 master_runner.cc:264] No leader master found from master e939ebcbbd4c4fc3b4da8246c17c2135
I20260620 01:55:34.383397 12351 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:55:34.389468 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 11909
I20260620 01:55:34.393715 12340 heartbeater.cc:499] Master 127.20.1.190:34393 was elected leader, sending a full tablet report...
I20260620 01:55:34.409076 12383 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0652566484b24e5cb7acc917520c310b" instance_seqno: 1781920533315957) as {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:49365; Asking this server to re-register.
I20260620 01:55:34.410729 12340 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:34.410723 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:38095
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--webserver_port=46275
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:55:34.413586 12383 ts_manager.cc:194] Registered new tserver with Master: 0652566484b24e5cb7acc917520c310b (127.20.1.131:35939)
I20260620 01:55:34.597112 12197 heartbeater.cc:344] Connected to a master server at 127.20.1.190:34393
I20260620 01:55:34.597492 12197 heartbeater.cc:499] Master 127.20.1.190:34393 was elected leader, sending a full tablet report...
I20260620 01:55:34.599412 12383 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f5b050223e074795b659b2c3b4273459" instance_seqno: 1781920532500764) as {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:57563; Asking this server to re-register.
I20260620 01:55:34.600544 12197 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:34.602662 12383 ts_manager.cc:194] Registered new tserver with Master: f5b050223e074795b659b2c3b4273459 (127.20.1.130:46165)
I20260620 01:55:34.832908 12445 raft_consensus.cc:493] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e939ebcbbd4c4fc3b4da8246c17c2135)
I20260620 01:55:34.833240 12445 raft_consensus.cc:515] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:34.835701 12445 leader_election.cc:290] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e939ebcbbd4c4fc3b4da8246c17c2135 (127.20.1.190:34393), 670ce53faa5041b5aebd1e588d7b1124 (127.20.1.189:36651)
I20260620 01:55:34.838764 11762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a892964204a44e5a8efd106641f0714a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "670ce53faa5041b5aebd1e588d7b1124" is_pre_election: true
I20260620 01:55:34.845849 12393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a892964204a44e5a8efd106641f0714a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" is_pre_election: true
I20260620 01:55:34.846768 12393 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a892964204a44e5a8efd106641f0714a in term 3.
I20260620 01:55:34.847612 11835 leader_election.cc:304] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a892964204a44e5a8efd106641f0714a, e939ebcbbd4c4fc3b4da8246c17c2135; no voters: 670ce53faa5041b5aebd1e588d7b1124
I20260620 01:55:34.848188 12445 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 01:55:34.848325 12445 raft_consensus.cc:493] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 3 FOLLOWER]: Starting leader election (detected failure of leader e939ebcbbd4c4fc3b4da8246c17c2135)
I20260620 01:55:34.848433 12445 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 3 FOLLOWER]: Advancing to term 4
I20260620 01:55:34.851220 12445 raft_consensus.cc:515] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:34.852417 12445 leader_election.cc:290] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [CANDIDATE]: Term 4 election: Requested vote from peers e939ebcbbd4c4fc3b4da8246c17c2135 (127.20.1.190:34393), 670ce53faa5041b5aebd1e588d7b1124 (127.20.1.189:36651)
I20260620 01:55:34.853161 11762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a892964204a44e5a8efd106641f0714a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "670ce53faa5041b5aebd1e588d7b1124"
I20260620 01:55:34.853163 12393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a892964204a44e5a8efd106641f0714a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135"
I20260620 01:55:34.853434 12393 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 3 FOLLOWER]: Advancing to term 4
I20260620 01:55:34.858036 12449 raft_consensus.cc:493] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e939ebcbbd4c4fc3b4da8246c17c2135)
I20260620 01:55:34.858559 12449 raft_consensus.cc:515] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:34.860762 12393 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a892964204a44e5a8efd106641f0714a in term 4.
I20260620 01:55:34.861011 12449 leader_election.cc:290] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e939ebcbbd4c4fc3b4da8246c17c2135 (127.20.1.190:34393), a892964204a44e5a8efd106641f0714a (127.20.1.188:39809)
I20260620 01:55:34.861447 11835 leader_election.cc:304] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a892964204a44e5a8efd106641f0714a, e939ebcbbd4c4fc3b4da8246c17c2135; no voters: 670ce53faa5041b5aebd1e588d7b1124
I20260620 01:55:34.862033 12445 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 4 FOLLOWER]: Leader election won for term 4
I20260620 01:55:34.867192 12445 raft_consensus.cc:697] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [term 4 LEADER]: Becoming Leader. State: Replica: a892964204a44e5a8efd106641f0714a, State: Running, Role: LEADER
I20260620 01:55:34.868012 12445 consensus_queue.cc:237] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } }
I20260620 01:55:34.872099 12393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "670ce53faa5041b5aebd1e588d7b1124" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" is_pre_election: true
I20260620 01:55:34.872676 12393 raft_consensus.cc:2393] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 670ce53faa5041b5aebd1e588d7b1124 in current term 4: Already voted for candidate a892964204a44e5a8efd106641f0714a in this term.
W20260620 01:55:34.872987 12443 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:34.873481 12443 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:34.873678 12443 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260620 01:55:34.874444 12453 sys_catalog.cc:455] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a892964204a44e5a8efd106641f0714a. Latest consensus state: current_term: 4 leader_uuid: "a892964204a44e5a8efd106641f0714a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:34.874804 12453 sys_catalog.cc:458] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: This master's current role is: LEADER
I20260620 01:55:34.875828 12455 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:55:34.879101 12455 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:55:34.879273 11860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "670ce53faa5041b5aebd1e588d7b1124" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a892964204a44e5a8efd106641f0714a" is_pre_election: true
I20260620 01:55:34.880522 12455 catalog_manager.cc:1294] Loaded cluster ID: a361767a6b444190b4b4e4a25e4ef9c4
I20260620 01:55:34.880472 11736 leader_election.cc:304] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 670ce53faa5041b5aebd1e588d7b1124; no voters: a892964204a44e5a8efd106641f0714a, e939ebcbbd4c4fc3b4da8246c17c2135
I20260620 01:55:34.880671 12455 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:55:34.881037 12449 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 3 FOLLOWER]: Advancing to term 4
I20260620 01:55:34.882580 12455 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:55:34.884017 12455 catalog_manager.cc:6080] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a: Loaded TSK: 0
I20260620 01:55:34.884188 12449 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 01:55:34.885670 12455 catalog_manager.cc:1549] Initializing in-progress tserver states...
W20260620 01:55:34.887862 12443 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:34.888067 12443 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:55:34.900825 12443 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:38095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=46275
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:34.902652 12443 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:34.904337 12443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:34.917323 12460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:34.917182 12458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:34.917173 12457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:34.919020 12443 server_base.cc:1061] running on GCE node
I20260620 01:55:34.920086 12443 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:34.920518 12443 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:34.920640 12443 hybrid_clock.cc:648] HybridClock initialized: now 1781920534920611 us; error 0 us; skew 500 ppm
I20260620 01:55:34.921061 12443 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:34 dist-test-slave-wg14 krb5kdc[11436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920534, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:34.926990 12443 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:55:34.929865 12443 webserver.cc:533] Webserver started at http://127.20.1.129:46275/ using document root <none> and password file <none>
I20260620 01:55:34.931165 12443 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:34.931391 12443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:34.939244 12443 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:55:34.944231 12466 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:34.947075 12443 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260620 01:55:34.947409 12443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "39486b58a61649dfababf1d4b5322a97"
format_stamp: "Formatted at 2026-06-20 01:55:08 on dist-test-slave-wg14"
I20260620 01:55:34.948288 12443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:34.967351 12443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:34.974134 12443 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:34.974457 12443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:34.975718 12443 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:55:34.977869 12443 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:55:34.978010 12443 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:34.978120 12443 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:55:34.978185 12443 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:35.028874 12443 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38095
I20260620 01:55:35.029098 12579 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38095 every 8 connection(s)
I20260620 01:55:35.031222 12443 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:55:35.038273 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 12443
Jun 20 01:55:35 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Jun 20 01:55:35 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920534, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:55:35 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920534, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:55:35 dist-test-slave-wg14 krb5kdc[11436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920534, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:55:35.071031 12582 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36651
I20260620 01:55:35.071521 12582 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:35.072657 12582 heartbeater.cc:507] Master 127.20.1.189:36651 requested a full tablet report, sending...
I20260620 01:55:35.073841 12581 heartbeater.cc:344] Connected to a master server at 127.20.1.188:39809
I20260620 01:55:35.074092 12581 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:35.074875 12581 heartbeater.cc:507] Master 127.20.1.188:39809 requested a full tablet report, sending...
I20260620 01:55:35.075969 11752 ts_manager.cc:194] Re-registered known tserver with Master: 39486b58a61649dfababf1d4b5322a97 (127.20.1.129:38095)
I20260620 01:55:35.077075 11848 ts_manager.cc:194] Re-registered known tserver with Master: 39486b58a61649dfababf1d4b5322a97 (127.20.1.129:38095)
I20260620 01:55:35.077927 12580 heartbeater.cc:344] Connected to a master server at 127.20.1.190:34393
I20260620 01:55:35.078140 12580 heartbeater.cc:461] Registering TS with master...
I20260620 01:55:35.078666 12580 heartbeater.cc:507] Master 127.20.1.190:34393 requested a full tablet report, sending...
I20260620 01:55:35.080233 11848 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:46439
I20260620 01:55:35.080639 12383 ts_manager.cc:194] Registered new tserver with Master: 39486b58a61649dfababf1d4b5322a97 (127.20.1.129:38095)
I20260620 01:55:35.235751 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 12443
W20260620 01:55:35.237483 12593 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:55:35.231771 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:56130 (local address 127.20.1.129:38095)
0620 01:55:35.233055 (+ 1284us) server_negotiation.cc:207] Beginning negotiation
0620 01:55:35.233065 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0620 01:55:35.233112 (+ 47us) server_negotiation.cc:408] Connection header received
0620 01:55:35.233202 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:55:35.233210 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:55:35.233488 (+ 278us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:55:35.233763 (+ 275us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:55:35.235547 (+ 1784us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56130: BlockingRecv error: recv got EOF from 127.0.0.1:56130 (error 108)
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":859,"threads_started":1}
W20260620 01:55:35.239234 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 12443: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:38095: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:55:35.239634 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 12443
I20260620 01:55:35.256965 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:38095
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--webserver_port=46275
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:55:35.289640 11762 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [term 4 FOLLOWER]: Refusing update from remote peer a892964204a44e5a8efd106641f0714a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260620 01:55:35.290643 12445 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [LEADER]: Connected to new peer: Peer: permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 01:55:35.293437 12449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a892964204a44e5a8efd106641f0714a. Latest consensus state: current_term: 4 leader_uuid: "a892964204a44e5a8efd106641f0714a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:35.293865 12449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:35.309468 12393 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [term 4 FOLLOWER]: Refusing update from remote peer a892964204a44e5a8efd106641f0714a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260620 01:55:35.310683 12453 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 01:55:35.312144 12449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "a892964204a44e5a8efd106641f0714a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:35.312445 12449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670ce53faa5041b5aebd1e588d7b1124 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:35.317674 12445 sys_catalog.cc:455] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "a892964204a44e5a8efd106641f0714a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:35.318137 12445 sys_catalog.cc:458] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: This master's current role is: LEADER
I20260620 01:55:35.328142 12599 sys_catalog.cc:455] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a892964204a44e5a8efd106641f0714a. Latest consensus state: current_term: 4 leader_uuid: "a892964204a44e5a8efd106641f0714a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:35.328575 12599 sys_catalog.cc:458] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:35.328951 12602 sys_catalog.cc:455] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "a892964204a44e5a8efd106641f0714a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:35.329334 12602 sys_catalog.cc:458] T 00000000000000000000000000000000 P e939ebcbbd4c4fc3b4da8246c17c2135 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:55:35.689796 12445 sys_catalog.cc:455] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "a892964204a44e5a8efd106641f0714a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e939ebcbbd4c4fc3b4da8246c17c2135" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34393 } } peers { permanent_uuid: "670ce53faa5041b5aebd1e588d7b1124" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36651 } } peers { permanent_uuid: "a892964204a44e5a8efd106641f0714a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39809 } } }
I20260620 01:55:35.690153 12445 sys_catalog.cc:458] T 00000000000000000000000000000000 P a892964204a44e5a8efd106641f0714a [sys.catalog]: This master's current role is: LEADER
W20260620 01:55:35.715879 12595 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:35.716260 12595 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:35.716379 12595 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:55:35.716442 12595 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:35.726766 12595 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:35.727047 12595 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:55:35.743044 12595 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:38095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=46275
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:35.745707 12595 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:35.747466 12595 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:35.760219 12609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:35.760946 12608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:35.761327 12611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:35.762858 12595 server_base.cc:1061] running on GCE node
I20260620 01:55:35.764334 12595 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:35.764583 12595 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:35.764709 12595 hybrid_clock.cc:648] HybridClock initialized: now 1781920535764664 us; error 0 us; skew 500 ppm
I20260620 01:55:35.765115 12595 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:35 dist-test-slave-wg14 krb5kdc[11436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920535, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:35.770469 12595 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:55:35.773664 12595 webserver.cc:533] Webserver started at http://127.20.1.129:46275/ using document root <none> and password file <none>
I20260620 01:55:35.775292 12595 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:35.775455 12595 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:35.785748 12595 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.008s sys 0.000s
I20260620 01:55:35.790438 12617 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:35.793423 12595 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:55:35.793874 12595 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "39486b58a61649dfababf1d4b5322a97"
format_stamp: "Formatted at 2026-06-20 01:55:08 on dist-test-slave-wg14"
I20260620 01:55:35.795032 12595 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:35.815829 12595 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:35.821468 12595 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:35.821820 12595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:35.822937 12595 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:55:35.825543 12595 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:55:35.825663 12595 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:35.825804 12595 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:55:35.825872 12595 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:55:35.826304 12595 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:55:35.826368 12595 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:55:35.826436 12595 fault_injection.cc:59] injecting fault for kudu (pid 12595): FLAGS_fault_before_start (process will exit)
I20260620 01:55:35.844241 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:38095
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--webserver_port=46275
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:55:36.306704 12625 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:55:36.307082 12625 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:55:36.307261 12625 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:55:36.318070 12625 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:55:36.318385 12625 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:55:36.334565 12625 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:38095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=46275
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:34393,127.20.1.189:36651,127.20.1.188:39809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:55:36.337311 12625 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:55:36.339725 12625 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:55:36.352608 12630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:36.353289 12633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:55:36.353510 12631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:55:36.354704 12625 server_base.cc:1061] running on GCE node
I20260620 01:55:36.355885 12625 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:55:36.356155 12625 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:55:36.356264 12625 hybrid_clock.cc:648] HybridClock initialized: now 1781920536356237 us; error 0 us; skew 500 ppm
I20260620 01:55:36.356639 12625 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:55:36 dist-test-slave-wg14 krb5kdc[11436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920536, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:55:36.361824 12625 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:55:36.364472 12625 webserver.cc:533] Webserver started at http://127.20.1.129:46275/ using document root <none> and password file <none>
I20260620 01:55:36.365433 12625 fs_manager.cc:362] Metadata directory not provided
I20260620 01:55:36.365581 12625 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:55:36.373159 12625 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 01:55:36.377822 12639 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:36.380052 12625 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:55:36.380363 12625 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "39486b58a61649dfababf1d4b5322a97"
format_stamp: "Formatted at 2026-06-20 01:55:08 on dist-test-slave-wg14"
I20260620 01:55:36.381387 12625 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:55:36.395843 12341 heartbeater.cc:499] Master 127.20.1.188:39809 was elected leader, sending a full tablet report...
I20260620 01:55:36.408782 12625 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:55:36.414237 12625 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:55:36.414597 12625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:55:36.415866 12625 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:55:36.418057 12625 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:55:36.418215 12625 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:36.418339 12625 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:55:36.418416 12625 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:55:36.466215 12625 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38095
I20260620 01:55:36.466640 12753 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38095 every 8 connection(s)
I20260620 01:55:36.468683 12625 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:55:36.471077 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 12625
I20260620 01:55:36.471616 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 12625
W20260620 01:55:36.497535 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 12625: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:38095: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:55:36.497759 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 12625
I20260620 01:55:36.512673 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 12069
W20260620 01:55:36.513746 12762 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:55:36.479212 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:36055 (local address 127.20.1.189:36651)
0620 01:55:36.489133 (+ 9921us) server_negotiation.cc:207] Beginning negotiation
0620 01:55:36.489143 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0620 01:55:36.494099 (+ 4956us) server_negotiation.cc:408] Connection header received
0620 01:55:36.495727 (+ 1628us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:55:36.495736 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:55:36.495838 (+ 102us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:55:36.495979 (+ 141us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:55:36.512475 (+ 16496us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:36055: BlockingRecv error: recv got EOF from 127.20.1.129:36055 (error 108)
Metrics: {"server-negotiator.queue_time_us":9748,"thread_start_us":9478,"threads_started":1}
W20260620 01:55:36.513748 12764 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:55:36.491428 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:35943 (local address 127.20.1.190:34393)
0620 01:55:36.492986 (+ 1558us) server_negotiation.cc:207] Beginning negotiation
0620 01:55:36.492994 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:55:36.494351 (+ 1357us) server_negotiation.cc:408] Connection header received
0620 01:55:36.494945 (+ 594us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:55:36.494952 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:55:36.495044 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:55:36.495272 (+ 228us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:55:36.512531 (+ 17259us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:35943: BlockingRecv error: recv got EOF from 127.20.1.129:35943 (error 108)
Metrics: {"server-negotiator.queue_time_us":1413,"thread_start_us":1259,"threads_started":1}
W20260620 01:55:36.513724 12759 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:55:36.472991 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:54191 (local address 127.20.1.188:39809)
0620 01:55:36.492749 (+ 19758us) server_negotiation.cc:207] Beginning negotiation
0620 01:55:36.492757 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:55:36.492790 (+ 33us) server_negotiation.cc:408] Connection header received
0620 01:55:36.495194 (+ 2404us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:55:36.495200 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:55:36.495311 (+ 111us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:55:36.495447 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:55:36.512500 (+ 17053us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:54191: BlockingRecv error: recv got EOF from 127.20.1.129:54191 (error 108)
Metrics: {"server-negotiator.queue_time_us":19574,"thread_start_us":19391,"threads_started":1}
W20260620 01:55:36.517576 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 12069: Not authorized: Client connection negotiation failed: client connection to 127.20.1.130:46165: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:55:36.517740 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 12069
I20260620 01:55:36.532590 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 12211
W20260620 01:55:36.536934 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 12211: Not authorized: Client connection negotiation failed: client connection to 127.20.1.131:35939: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:55:36.537142 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 12211
I20260620 01:55:36.551391 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 12351
W20260620 01:55:36.554777 12764 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:55:36.552583 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.20.1.190:34393)
0620 01:55:36.552834 (+ 251us) server_negotiation.cc:207] Beginning negotiation
0620 01:55:36.552839 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0620 01:55:36.552868 (+ 29us) server_negotiation.cc:408] Connection header received
0620 01:55:36.552943 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:55:36.552948 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:55:36.553034 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:55:36.553140 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:55:36.554345 (+ 1205us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48598: BlockingRecv error: recv got EOF from 127.0.0.1:48598 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
W20260620 01:55:36.554710 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 12351: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:34393: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:55:36.555104 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 12351
I20260620 01:55:36.575897 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 11721
I20260620 01:55:36.577553 11783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:55:36.707376 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 11721
I20260620 01:55:36.731705 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 11811
I20260620 01:55:36.733083 11880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:55:36.878361 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 11811
W20260620 01:55:36.878271 11835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.190:34393: connect: Connection refused (error 111)
Shutting down hive metastore.
[ OK ] ExternalMiniClusterTest.TestBasicOperation/9 (57311 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/10
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 01:55:37 dist-test-slave-wg14 krb5kdc[12777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 01:55:37 dist-test-slave-wg14 krb5kdc[12777](info): set up 2 sockets
Jun 20 01:55:37 dist-test-slave-wg14 krb5kdc[12777](info): commencing operation
krb5kdc: starting...
W20260620 01:55:39.427379 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.108s user 0.002s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 01:55:39 dist-test-slave-wg14 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T01:55:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:55:39Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf
2026-06-20 01:55:49: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Jun 20 01:55:52 dist-test-slave-wg14 krb5kdc[12777](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920552, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 01:55:59 dist-test-slave-wg14 krb5kdc[12777](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920559, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 01:55:59 dist-test-slave-wg14 krb5kdc[12777](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920559, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:56:00.109599 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:45553
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:56:00.523648 13079 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:00.524046 13079 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:00.524185 13079 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:00.533629 13079 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:56:00.533811 13079 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:00.533890 13079 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:56:00.533952 13079 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:56:00.546144 13079 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:45553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:00.548105 13079 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:00.549612 13079 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:00.560331 13084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:00.560703 13085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:00.562037 13087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:00.563535 13079 server_base.cc:1061] running on GCE node
I20260620 01:56:00.564730 13079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:00.567085 13079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:00.568689 13079 hybrid_clock.cc:648] HybridClock initialized: now 1781920560568447 us; error 66 us; skew 500 ppm
I20260620 01:56:00.569628 13079 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:00 dist-test-slave-wg14 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920560, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:00.574518 13079 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:56:00.576913 13079 webserver.cc:533] Webserver started at http://127.20.1.190:45093/ using document root <none> and password file <none>
I20260620 01:56:00.577838 13079 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:00.577991 13079 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:00.578470 13079 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:56:00.582420 13079 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "f7400b9832ac44a1ad0dc8f8f101ea98"
format_stamp: "Formatted at 2026-06-20 01:56:00 on dist-test-slave-wg14"
I20260620 01:56:00.584075 13079 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "f7400b9832ac44a1ad0dc8f8f101ea98"
format_stamp: "Formatted at 2026-06-20 01:56:00 on dist-test-slave-wg14"
I20260620 01:56:00.593914 13079 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260620 01:56:00.600050 13094 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:00.602677 13079 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:56:00.603621 13079 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "f7400b9832ac44a1ad0dc8f8f101ea98"
format_stamp: "Formatted at 2026-06-20 01:56:00 on dist-test-slave-wg14"
I20260620 01:56:00.604013 13079 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:00.649501 13079 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:00.659822 13079 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:00.660175 13079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:00.685798 13079 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:45553
I20260620 01:56:00.685819 13145 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:45553 every 8 connection(s)
I20260620 01:56:00.687603 13079 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:56:00.688269 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 13079
I20260620 01:56:00.688488 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:56:00.693748 13146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:56:00.701781 13146 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.189@KRBTEST.COM" created.
Jun 20 01:56:00 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920560, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:56:00.736690 13146 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:56:00.739050 13096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:33725: connect: Connection refused (error 111)
W20260620 01:56:00.741910 13146 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:33725: Network error: Client connection negotiation failed: client connection to 127.20.1.189:33725: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.189@KRBTEST.COM" created.
I20260620 01:56:00.791482 13146 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } attempt: 1
W20260620 01:56:00.795176 13146 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:33725: Network error: Client connection negotiation failed: client connection to 127.20.1.189:33725: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:56:00.816258 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:33725
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:56:00.863564 13146 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } attempt: 2
W20260620 01:56:00.866708 13146 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:33725: Network error: Client connection negotiation failed: client connection to 127.20.1.189:33725: connect: Connection refused (error 111)
I20260620 01:56:00.958151 13146 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } attempt: 3
W20260620 01:56:00.961403 13146 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:33725: Network error: Client connection negotiation failed: client connection to 127.20.1.189:33725: connect: Connection refused (error 111)
I20260620 01:56:01.104779 13146 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } attempt: 4
W20260620 01:56:01.107287 13146 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:33725: Network error: Client connection negotiation failed: client connection to 127.20.1.189:33725: connect: Connection refused (error 111)
W20260620 01:56:01.302026 13156 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:01.302438 13156 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:01.302574 13156 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:01.312557 13156 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:56:01.312728 13156 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:01.312811 13156 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:56:01.312881 13156 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:56:01.326380 13156 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:33725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:01.328305 13156 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:01.330179 13156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:01.341583 13161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:01.341614 13162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:01.341614 13164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:01.343892 13156 server_base.cc:1061] running on GCE node
I20260620 01:56:01.345324 13156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:01.347266 13156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:01.348594 13156 hybrid_clock.cc:648] HybridClock initialized: now 1781920561348537 us; error 62 us; skew 500 ppm
I20260620 01:56:01.349071 13156 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:01 dist-test-slave-wg14 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920561, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:01.353515 13156 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 01:56:01.355988 13156 webserver.cc:533] Webserver started at http://127.20.1.189:33659/ using document root <none> and password file <none>
I20260620 01:56:01.357086 13156 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:01.357245 13156 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:01.357684 13156 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:56:01.360637 13156 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "d5f2999892d543c2a5e963a26648f971"
format_stamp: "Formatted at 2026-06-20 01:56:01 on dist-test-slave-wg14"
I20260620 01:56:01.361593 13156 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "d5f2999892d543c2a5e963a26648f971"
format_stamp: "Formatted at 2026-06-20 01:56:01 on dist-test-slave-wg14"
I20260620 01:56:01.369879 13156 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.003s
I20260620 01:56:01.375103 13171 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:01.377332 13156 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260620 01:56:01.377651 13156 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "d5f2999892d543c2a5e963a26648f971"
format_stamp: "Formatted at 2026-06-20 01:56:01 on dist-test-slave-wg14"
I20260620 01:56:01.377938 13156 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:01.406607 13146 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } attempt: 5
W20260620 01:56:01.409227 13146 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:33725: Network error: Client connection negotiation failed: client connection to 127.20.1.189:33725: connect: Connection refused (error 111)
I20260620 01:56:01.413264 13156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:01.418316 13156 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:01.418679 13156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:01.444116 13156 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:33725
I20260620 01:56:01.444128 13222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:33725 every 8 connection(s)
I20260620 01:56:01.446503 13156 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:56:01.452062 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 13156
I20260620 01:56:01.452310 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:56:01.453320 13223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:56:01.461562 13223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
Jun 20 01:56:01 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920561, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.188@KRBTEST.COM" created.
I20260620 01:56:01.491962 13223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:56:01 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920561, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:56:01.519310 13223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:56:01.521147 13174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:42201: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:56:01.524115 13223 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:42201: Network error: Client connection negotiation failed: client connection to 127.20.1.188:42201: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.188@KRBTEST.COM" created.
I20260620 01:56:01.573639 13223 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } attempt: 1
W20260620 01:56:01.577230 13223 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:42201: Network error: Client connection negotiation failed: client connection to 127.20.1.188:42201: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:56:01.600654 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:42201
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:56:01.645642 13223 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } attempt: 2
W20260620 01:56:01.649592 13223 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:42201: Network error: Client connection negotiation failed: client connection to 127.20.1.188:42201: connect: Connection refused (error 111)
I20260620 01:56:01.741010 13223 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } attempt: 3
W20260620 01:56:01.744233 13223 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:42201: Network error: Client connection negotiation failed: client connection to 127.20.1.188:42201: connect: Connection refused (error 111)
I20260620 01:56:01.887580 13223 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } attempt: 4
W20260620 01:56:01.890666 13223 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:42201: Network error: Client connection negotiation failed: client connection to 127.20.1.188:42201: connect: Connection refused (error 111)
I20260620 01:56:01.956565 13146 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } attempt: 6
Jun 20 01:56:01 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920560, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:56:01.981774 13146 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:56:01.985908 13146 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:42201: Network error: Client connection negotiation failed: client connection to 127.20.1.188:42201: connect: Connection refused (error 111)
I20260620 01:56:02.038370 13146 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } attempt: 1
W20260620 01:56:02.042233 13146 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:42201: Network error: Client connection negotiation failed: client connection to 127.20.1.188:42201: connect: Connection refused (error 111)
W20260620 01:56:02.056933 13233 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:02.057322 13233 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:02.057412 13233 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:02.066969 13233 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:56:02.067112 13233 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:02.067227 13233 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:56:02.067273 13233 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:56:02.080278 13233 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:42201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:02.082858 13233 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:02.085595 13233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:02.098567 13240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:02.098553 13241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:02.099187 13243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:02.099260 13233 server_base.cc:1061] running on GCE node
I20260620 01:56:02.100423 13233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:02.102916 13233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:02.104568 13233 hybrid_clock.cc:648] HybridClock initialized: now 1781920562104466 us; error 111 us; skew 500 ppm
I20260620 01:56:02.105198 13233 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:02 dist-test-slave-wg14 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920562, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:02.109779 13233 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 01:56:02.112892 13233 webserver.cc:533] Webserver started at http://127.20.1.188:37427/ using document root <none> and password file <none>
I20260620 01:56:02.114043 13233 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:02.114225 13233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:02.114652 13233 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:56:02.116559 13146 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } attempt: 2
I20260620 01:56:02.118357 13233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "1010357ab4514fdea43e3a6cf7d9ac9c"
format_stamp: "Formatted at 2026-06-20 01:56:02 on dist-test-slave-wg14"
W20260620 01:56:02.119709 13146 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:42201: Network error: Client connection negotiation failed: client connection to 127.20.1.188:42201: connect: Connection refused (error 111)
I20260620 01:56:02.119789 13233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "1010357ab4514fdea43e3a6cf7d9ac9c"
format_stamp: "Formatted at 2026-06-20 01:56:02 on dist-test-slave-wg14"
I20260620 01:56:02.129113 13233 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260620 01:56:02.134503 13250 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:02.137396 13233 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:56:02.137678 13233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "1010357ab4514fdea43e3a6cf7d9ac9c"
format_stamp: "Formatted at 2026-06-20 01:56:02 on dist-test-slave-wg14"
I20260620 01:56:02.137918 13233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:02.162103 13233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:02.168736 13233 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:02.169103 13233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:02.189993 13223 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } attempt: 5
W20260620 01:56:02.193356 13223 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:42201: Network error: Client connection negotiation failed: client connection to 127.20.1.188:42201: connect: Connection refused (error 111)
I20260620 01:56:02.194169 13233 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:42201
I20260620 01:56:02.194316 13302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:42201 every 8 connection(s)
I20260620 01:56:02.196199 13233 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:56:02.203266 13303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:56:02.205832 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 13233
I20260620 01:56:02.206099 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:56:02.214778 13303 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:56:02 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:56:02.233034 13146 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } attempt: 3
Jun 20 01:56:02 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920562, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:56:02 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:56:02 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920560, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:56:02.248030 13303 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:56:02 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920562, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:56:02 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:56:02.266114 13303 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:56:02.268965 13146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Bootstrap starting.
Jun 20 01:56:02 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920562, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:56:02.274847 13146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Neither blocks nor log segments found. Creating new log.
I20260620 01:56:02.277172 13146 log.cc:826] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Log is configured to *not* fsync() on all Append() calls
I20260620 01:56:02.280946 13146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: No bootstrap required, opened a new log
I20260620 01:56:02.290184 13146 raft_consensus.cc:359] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:02.290518 13146 raft_consensus.cc:385] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:56:02.290622 13146 raft_consensus.cc:740] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7400b9832ac44a1ad0dc8f8f101ea98, State: Initialized, Role: FOLLOWER
I20260620 01:56:02.291174 13303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c: Bootstrap starting.
I20260620 01:56:02.291421 13146 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:02.292778 13313 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:02.293358 13313 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:02.294389 13146 sys_catalog.cc:565] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:56:02.297725 13303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c: Neither blocks nor log segments found. Creating new log.
I20260620 01:56:02.299496 13303 log.cc:826] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c: Log is configured to *not* fsync() on all Append() calls
I20260620 01:56:02.303257 13303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c: No bootstrap required, opened a new log
I20260620 01:56:02.311437 13303 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
W20260620 01:56:02.312074 13325 catalog_manager.cc:1593] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:56:02.312302 13325 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:56:02.312433 13303 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:56:02.312635 13303 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1010357ab4514fdea43e3a6cf7d9ac9c, State: Initialized, Role: FOLLOWER
I20260620 01:56:02.313802 13303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:02.315516 13327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:02.309433 13079 master_runner.cc:187] Error getting master registration for 127.20.1.189:33725: OK, instance_id { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" instance_seqno: 1781920561436644 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:56:02.316190 13327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:02.316308 13079 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:56:02.317796 13303 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:56:02.336941 13338 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:56:02.337381 13338 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:56:02.333863 13233 master_runner.cc:187] Error getting master registration for 127.20.1.189:33725: OK, instance_id { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" instance_seqno: 1781920561436644 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:56:02.342270 13233 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:56:02.363318 13313 raft_consensus.cc:493] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:56:02.363799 13313 raft_consensus.cc:515] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:02.366174 13313 leader_election.cc:290] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5f2999892d543c2a5e963a26648f971 (127.20.1.189:33725), 1010357ab4514fdea43e3a6cf7d9ac9c (127.20.1.188:42201)
I20260620 01:56:02.367266 13277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" is_pre_election: true
I20260620 01:56:02.368090 13277 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7400b9832ac44a1ad0dc8f8f101ea98 in term 0.
I20260620 01:56:02.369501 13097 leader_election.cc:304] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1010357ab4514fdea43e3a6cf7d9ac9c, f7400b9832ac44a1ad0dc8f8f101ea98; no voters:
I20260620 01:56:02.371116 13313 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:56:02.371385 13313 raft_consensus.cc:493] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:56:02.371556 13313 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:56:02.376281 13313 raft_consensus.cc:515] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:02.378723 13277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c"
I20260620 01:56:02.379213 13277 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:56:02.384166 13277 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7400b9832ac44a1ad0dc8f8f101ea98 in term 1.
I20260620 01:56:02.384948 13097 leader_election.cc:304] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1010357ab4514fdea43e3a6cf7d9ac9c, f7400b9832ac44a1ad0dc8f8f101ea98; no voters:
I20260620 01:56:02.391105 13313 leader_election.cc:290] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [CANDIDATE]: Term 1 election: Requested vote from peers d5f2999892d543c2a5e963a26648f971 (127.20.1.189:33725), 1010357ab4514fdea43e3a6cf7d9ac9c (127.20.1.188:42201)
I20260620 01:56:02.392503 13313 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:56:02.392722 13313 raft_consensus.cc:697] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 1 LEADER]: Becoming Leader. State: Replica: f7400b9832ac44a1ad0dc8f8f101ea98, State: Running, Role: LEADER
I20260620 01:56:02.393421 13313 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
W20260620 01:56:02.414932 13100 tablet.cc:2406] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:56:02.417407 13340 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7400b9832ac44a1ad0dc8f8f101ea98. Latest consensus state: current_term: 1 leader_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:02.419243 13340 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: This master's current role is: LEADER
W20260620 01:56:02.422328 13256 tablet.cc:2406] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:56:02.424785 13342 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:56:02.434558 13342 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:56:02.451735 13277 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 1 FOLLOWER]: Refusing update from remote peer f7400b9832ac44a1ad0dc8f8f101ea98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:56:02.456329 13340 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:56:02.473682 13327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7400b9832ac44a1ad0dc8f8f101ea98. Latest consensus state: current_term: 1 leader_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:02.473991 13327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:02.491714 13313 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:02.492146 13313 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: This master's current role is: LEADER
I20260620 01:56:02.496636 13327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:02.496896 13327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:02.497808 13342 catalog_manager.cc:1382] Generated new cluster ID: 582c0e6637124b4fad99bb90819d0134
I20260620 01:56:02.497967 13342 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:56:02.527064 13342 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:56:02.529609 13342 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:56:02.562565 13342 catalog_manager.cc:6069] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Generated new TSK 0
I20260620 01:56:02.569047 13342 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:56:02.695453 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.104s user 0.010s sys 0.019s
I20260620 01:56:02.740691 13223 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } attempt: 6
Jun 20 01:56:02 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920561, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
I20260620 01:56:02.807291 13223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971: Bootstrap starting.
I20260620 01:56:02.817505 13223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971: Neither blocks nor log segments found. Creating new log.
I20260620 01:56:02.823458 13223 log.cc:826] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971: Log is configured to *not* fsync() on all Append() calls
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:56:02.831352 13223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971: No bootstrap required, opened a new log
W20260620 01:56:02.831724 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.136s user 0.000s sys 0.025s
I20260620 01:56:02.833616 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:56:02.854880 13223 raft_consensus.cc:359] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:02.855520 13223 raft_consensus.cc:385] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:56:02.855680 13223 raft_consensus.cc:740] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f2999892d543c2a5e963a26648f971, State: Initialized, Role: FOLLOWER
I20260620 01:56:02.856755 13223 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:02.864253 13355 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:02.864712 13355 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:02.868613 13223 sys_catalog.cc:565] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:56:02.868956 13198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f2999892d543c2a5e963a26648f971" is_pre_election: true
I20260620 01:56:02.869781 13198 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7400b9832ac44a1ad0dc8f8f101ea98 in term 0.
I20260620 01:56:02.870326 13197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f2999892d543c2a5e963a26648f971"
I20260620 01:56:02.870653 13197 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:56:02.877794 13197 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7400b9832ac44a1ad0dc8f8f101ea98 in term 1.
I20260620 01:56:02.879161 13196 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 1 FOLLOWER]: Refusing update from remote peer f7400b9832ac44a1ad0dc8f8f101ea98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:56:02.881954 13313 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:56:02.940662 13355 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7400b9832ac44a1ad0dc8f8f101ea98. Latest consensus state: current_term: 1 leader_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:02.940737 13364 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:02.941308 13355 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:02.941501 13364 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:02.983867 13371 catalog_manager.cc:1294] Loaded cluster ID: 582c0e6637124b4fad99bb90819d0134
I20260620 01:56:02.984050 13371 catalog_manager.cc:1587] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971: loading cluster ID for follower catalog manager: success
I20260620 01:56:02.989944 13371 catalog_manager.cc:1609] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971: acquiring CA information for follower catalog manager: success
I20260620 01:56:02.994398 13371 catalog_manager.cc:1637] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:56:03.340818 13338 catalog_manager.cc:1294] Loaded cluster ID: 582c0e6637124b4fad99bb90819d0134
I20260620 01:56:03.341049 13338 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c: loading cluster ID for follower catalog manager: success
I20260620 01:56:03.346235 13338 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c: acquiring CA information for follower catalog manager: success
I20260620 01:56:03.348800 13338 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:56:03.508783 13378 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:03.509413 13378 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: This master's current role is: LEADER
W20260620 01:56:03.511276 13354 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:03.511831 13354 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:03.512043 13354 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:03.525969 13354 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:03.526237 13354 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:56:03.541934 13354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:03.544133 13354 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:03.546051 13354 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:03.559521 13383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:03.559465 13382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:03.561735 13385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:03.562335 13354 server_base.cc:1061] running on GCE node
I20260620 01:56:03.563212 13354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:03.565292 13354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:03.566856 13354 hybrid_clock.cc:648] HybridClock initialized: now 1781920563566752 us; error 103 us; skew 500 ppm
I20260620 01:56:03.567337 13354 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:03 dist-test-slave-wg14 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920563, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:03.572266 13354 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:56:03.574983 13354 webserver.cc:533] Webserver started at http://127.20.1.129:34163/ using document root <none> and password file <none>
I20260620 01:56:03.576025 13354 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:03.576176 13354 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:03.576650 13354 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:56:03.580643 13354 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "317a641b28a547f9a076cf94c18c114a"
format_stamp: "Formatted at 2026-06-20 01:56:03 on dist-test-slave-wg14"
I20260620 01:56:03.581916 13354 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "317a641b28a547f9a076cf94c18c114a"
format_stamp: "Formatted at 2026-06-20 01:56:03 on dist-test-slave-wg14"
I20260620 01:56:03.591025 13354 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.004s
I20260620 01:56:03.595844 13392 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:03.598207 13354 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260620 01:56:03.598465 13354 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "317a641b28a547f9a076cf94c18c114a"
format_stamp: "Formatted at 2026-06-20 01:56:03 on dist-test-slave-wg14"
I20260620 01:56:03.598735 13354 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:03.626019 13354 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:03.632390 13354 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:03.632740 13354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:03.633903 13354 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:56:03.635957 13354 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:56:03.636072 13354 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:03.636197 13354 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:56:03.636252 13354 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:03.678579 13505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35175 every 8 connection(s)
I20260620 01:56:03.678668 13354 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35175
I20260620 01:56:03.681190 13354 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:56:03.685508 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 13354
I20260620 01:56:03.685760 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 01:56:03 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920563, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:56:03 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920563, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:56:03 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920563, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.130@KRBTEST.COM" created.
I20260620 01:56:03.736402 13508 heartbeater.cc:344] Connected to a master server at 127.20.1.189:33725
I20260620 01:56:03.736944 13508 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:03.737931 13508 heartbeater.cc:507] Master 127.20.1.189:33725 requested a full tablet report, sending...
I20260620 01:56:03.738936 13506 heartbeater.cc:344] Connected to a master server at 127.20.1.190:45553
I20260620 01:56:03.739163 13506 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:03.739823 13506 heartbeater.cc:507] Master 127.20.1.190:45553 requested a full tablet report, sending...
I20260620 01:56:03.740808 13507 heartbeater.cc:344] Connected to a master server at 127.20.1.188:42201
I20260620 01:56:03.741060 13507 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:03.741762 13507 heartbeater.cc:507] Master 127.20.1.188:42201 requested a full tablet report, sending...
I20260620 01:56:03.741868 13188 ts_manager.cc:194] Registered new tserver with Master: 317a641b28a547f9a076cf94c18c114a (127.20.1.129:35175)
I20260620 01:56:03.741887 13110 ts_manager.cc:194] Registered new tserver with Master: 317a641b28a547f9a076cf94c18c114a (127.20.1.129:35175)
I20260620 01:56:03.744951 13267 ts_manager.cc:194] Registered new tserver with Master: 317a641b28a547f9a076cf94c18c114a (127.20.1.129:35175)
I20260620 01:56:03.744952 13110 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:53815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:56:03.842893 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:56:04.293704 13520 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:04.294137 13520 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:04.294307 13520 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:04.304610 13520 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:04.304984 13520 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:56:04.319638 13520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:04.321938 13520 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:04.323880 13520 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:04.340657 13527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:04.340622 13526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:04.341523 13529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:04.342195 13520 server_base.cc:1061] running on GCE node
I20260620 01:56:04.343236 13520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:04.345865 13520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:04.347337 13520 hybrid_clock.cc:648] HybridClock initialized: now 1781920564347238 us; error 71 us; skew 500 ppm
I20260620 01:56:04.347884 13520 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:04 dist-test-slave-wg14 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920564, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:04.353713 13520 init.cc:377] Logged in from keytab as kudu/127.20.1.130@KRBTEST.COM (short username kudu)
I20260620 01:56:04.357576 13520 webserver.cc:533] Webserver started at http://127.20.1.130:38257/ using document root <none> and password file <none>
I20260620 01:56:04.359318 13520 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:04.359484 13520 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:04.360373 13520 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:56:04.365229 13520 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/data/instance:
uuid: "733436d44f6343cdb1325bf4d4531311"
format_stamp: "Formatted at 2026-06-20 01:56:04 on dist-test-slave-wg14"
I20260620 01:56:04.367113 13520 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance:
uuid: "733436d44f6343cdb1325bf4d4531311"
format_stamp: "Formatted at 2026-06-20 01:56:04 on dist-test-slave-wg14"
I20260620 01:56:04.379245 13520 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.009s sys 0.000s
I20260620 01:56:04.385659 13537 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:04.388538 13520 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:56:04.388849 13520 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "733436d44f6343cdb1325bf4d4531311"
format_stamp: "Formatted at 2026-06-20 01:56:04 on dist-test-slave-wg14"
I20260620 01:56:04.389163 13520 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:04.435246 13520 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:04.440769 13520 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:04.441359 13520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:04.443066 13520 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:56:04.446002 13520 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:56:04.446220 13520 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:04.446405 13520 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:56:04.446475 13520 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:04.489236 13520 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:46557
I20260620 01:56:04.489271 13650 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:46557 every 8 connection(s)
I20260620 01:56:04.491288 13520 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:56:04.499004 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 13520
I20260620 01:56:04.499476 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
Jun 20 01:56:04 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920564, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:56:04 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920564, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:56:04 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920564, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.131@KRBTEST.COM; defaulting to no policy
I20260620 01:56:04.548440 13653 heartbeater.cc:344] Connected to a master server at 127.20.1.189:33725
I20260620 01:56:04.548770 13652 heartbeater.cc:344] Connected to a master server at 127.20.1.188:42201
I20260620 01:56:04.549027 13653 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:04.549154 13652 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:04.550606 13652 heartbeater.cc:507] Master 127.20.1.188:42201 requested a full tablet report, sending...
I20260620 01:56:04.550628 13653 heartbeater.cc:507] Master 127.20.1.189:33725 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.131@KRBTEST.COM" created.
I20260620 01:56:04.553314 13188 ts_manager.cc:194] Registered new tserver with Master: 733436d44f6343cdb1325bf4d4531311 (127.20.1.130:46557)
I20260620 01:56:04.553699 13267 ts_manager.cc:194] Registered new tserver with Master: 733436d44f6343cdb1325bf4d4531311 (127.20.1.130:46557)
I20260620 01:56:04.558315 13651 heartbeater.cc:344] Connected to a master server at 127.20.1.190:45553
I20260620 01:56:04.558519 13651 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:04.559042 13651 heartbeater.cc:507] Master 127.20.1.190:45553 requested a full tablet report, sending...
I20260620 01:56:04.560425 13110 ts_manager.cc:194] Registered new tserver with Master: 733436d44f6343cdb1325bf4d4531311 (127.20.1.130:46557)
I20260620 01:56:04.561357 13110 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:33409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:56:04.659643 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:56:04.748208 13506 heartbeater.cc:499] Master 127.20.1.190:45553 was elected leader, sending a full tablet report...
W20260620 01:56:05.117172 13665 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:05.117539 13665 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:05.117673 13665 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:05.127852 13665 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:05.128065 13665 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:56:05.143674 13665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:05.146142 13665 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:05.148404 13665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:05.163290 13670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:05.163352 13671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:05.163393 13673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:05.164762 13665 server_base.cc:1061] running on GCE node
I20260620 01:56:05.165671 13665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:05.166903 13665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:05.168218 13665 hybrid_clock.cc:648] HybridClock initialized: now 1781920565168157 us; error 78 us; skew 500 ppm
I20260620 01:56:05.168620 13665 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:05 dist-test-slave-wg14 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920565, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:05.173225 13665 init.cc:377] Logged in from keytab as kudu/127.20.1.131@KRBTEST.COM (short username kudu)
I20260620 01:56:05.175397 13665 webserver.cc:533] Webserver started at http://127.20.1.131:45879/ using document root <none> and password file <none>
I20260620 01:56:05.176316 13665 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:05.176448 13665 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:05.176855 13665 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:56:05.180027 13665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/data/instance:
uuid: "917d6f4589cd4b4c9d9b30ac6d3b8aa9"
format_stamp: "Formatted at 2026-06-20 01:56:05 on dist-test-slave-wg14"
I20260620 01:56:05.181149 13665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance:
uuid: "917d6f4589cd4b4c9d9b30ac6d3b8aa9"
format_stamp: "Formatted at 2026-06-20 01:56:05 on dist-test-slave-wg14"
I20260620 01:56:05.189049 13665 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260620 01:56:05.194705 13680 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:05.198173 13665 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260620 01:56:05.198467 13665 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "917d6f4589cd4b4c9d9b30ac6d3b8aa9"
format_stamp: "Formatted at 2026-06-20 01:56:05 on dist-test-slave-wg14"
I20260620 01:56:05.198846 13665 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:05.229348 13665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:05.235351 13665 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:05.235668 13665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:05.236876 13665 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:56:05.239044 13665 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:56:05.239179 13665 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 01:56:05.239300 13665 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:56:05.239377 13665 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:05.291296 13665 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:44667
I20260620 01:56:05.291378 13793 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:44667 every 8 connection(s)
I20260620 01:56:05.294454 13665 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:56:05.300678 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 13665
I20260620 01:56:05.300963 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
Jun 20 01:56:05 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920565, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:56:05 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920565, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:56:05 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920565, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:56:05.339712 13796 heartbeater.cc:344] Connected to a master server at 127.20.1.189:33725
I20260620 01:56:05.340158 13796 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:05.342065 13796 heartbeater.cc:507] Master 127.20.1.189:33725 requested a full tablet report, sending...
I20260620 01:56:05.344169 13794 heartbeater.cc:344] Connected to a master server at 127.20.1.190:45553
I20260620 01:56:05.344249 13795 heartbeater.cc:344] Connected to a master server at 127.20.1.188:42201
I20260620 01:56:05.344347 13188 ts_manager.cc:194] Registered new tserver with Master: 917d6f4589cd4b4c9d9b30ac6d3b8aa9 (127.20.1.131:44667)
I20260620 01:56:05.344379 13794 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:05.344414 13795 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:05.345027 13794 heartbeater.cc:507] Master 127.20.1.190:45553 requested a full tablet report, sending...
I20260620 01:56:05.345048 13795 heartbeater.cc:507] Master 127.20.1.188:42201 requested a full tablet report, sending...
I20260620 01:56:05.346338 13110 ts_manager.cc:194] Registered new tserver with Master: 917d6f4589cd4b4c9d9b30ac6d3b8aa9 (127.20.1.131:44667)
I20260620 01:56:05.346592 13267 ts_manager.cc:194] Registered new tserver with Master: 917d6f4589cd4b4c9d9b30ac6d3b8aa9 (127.20.1.131:44667)
I20260620 01:56:05.348094 13110 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:56613
I20260620 01:56:05.358970 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:56:05.372148 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:56:05.372372 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 13079
I20260620 01:56:05.373950 13141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:56:05.509048 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 13079
W20260620 01:56:05.530591 13175 connection.cc:573] server connection from 127.0.0.1:60172 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:56:05.530967 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:45553
--webserver_interface=127.20.1.190
--webserver_port=45093
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:56:05.531014 13254 connection.cc:573] server connection from 127.0.0.1:57796 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:56:05.565519 13651 heartbeater.cc:499] Master 127.20.1.190:45553 was elected leader, sending a full tablet report...
W20260620 01:56:05.567706 13651 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:45553 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:45553: connect: Connection refused (error 111)
W20260620 01:56:05.752678 13506 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:45553 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:45553: connect: Connection refused (error 111)
W20260620 01:56:05.999397 13806 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:05.999897 13806 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:06.000052 13806 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:06.011085 13806 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:56:06.011271 13806 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:06.011351 13806 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:56:06.011415 13806 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:56:06.027045 13806 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:45553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=45093
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:06.029160 13806 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:06.031296 13806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:06.042127 13813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:06.043103 13814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:06.044454 13816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:06.045588 13806 server_base.cc:1061] running on GCE node
I20260620 01:56:06.046569 13806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:06.048764 13806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:06.050159 13806 hybrid_clock.cc:648] HybridClock initialized: now 1781920566050075 us; error 70 us; skew 500 ppm
I20260620 01:56:06.050800 13806 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:06 dist-test-slave-wg14 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920566, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:06.056684 13806 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:56:06.060771 13806 webserver.cc:533] Webserver started at http://127.20.1.190:45093/ using document root <none> and password file <none>
I20260620 01:56:06.061791 13806 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:06.061972 13806 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:06.070058 13806 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:56:06.074743 13823 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:06.077491 13806 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:56:06.077783 13806 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "f7400b9832ac44a1ad0dc8f8f101ea98"
format_stamp: "Formatted at 2026-06-20 01:56:00 on dist-test-slave-wg14"
I20260620 01:56:06.078953 13806 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:06.098129 13806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:06.109241 13806 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:06.109608 13806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:06.136777 13806 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:45553
I20260620 01:56:06.136845 13874 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:45553 every 8 connection(s)
I20260620 01:56:06.138768 13806 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:56:06.147989 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 13806
I20260620 01:56:06.148509 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 13354
I20260620 01:56:06.150295 13875 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:56:06.155784 13875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Bootstrap starting.
Jun 20 01:56:06 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 01:56:06.172487 13501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:56:06.179642 13875 log.cc:826] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Log is configured to *not* fsync() on all Append() calls
I20260620 01:56:06.197928 13875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:56:06.198765 13875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Bootstrap complete.
I20260620 01:56:06.211491 13875 raft_consensus.cc:359] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:06.212770 13875 raft_consensus.cc:740] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7400b9832ac44a1ad0dc8f8f101ea98, State: Initialized, Role: FOLLOWER
I20260620 01:56:06.213986 13875 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:06.215189 13881 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:06.215494 13881 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:06.216825 13875 sys_catalog.cc:565] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:56:06.238682 13892 catalog_manager.cc:1294] Loaded cluster ID: 582c0e6637124b4fad99bb90819d0134
I20260620 01:56:06.238847 13892 catalog_manager.cc:1587] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: loading cluster ID for follower catalog manager: success
I20260620 01:56:06.244592 13892 catalog_manager.cc:1609] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: acquiring CA information for follower catalog manager: success
I20260620 01:56:06.248561 13892 catalog_manager.cc:1637] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Jun 20 01:56:06 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920566, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:56:06.312183 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 13354
I20260620 01:56:06.323827 13806 master_runner.cc:264] No leader master found from master f7400b9832ac44a1ad0dc8f8f101ea98
I20260620 01:56:06.324046 13806 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:56:06.332417 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35175
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--webserver_port=34163
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:56:06.352573 13794 heartbeater.cc:499] Master 127.20.1.190:45553 was elected leader, sending a full tablet report...
I20260620 01:56:06.366050 13840 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "917d6f4589cd4b4c9d9b30ac6d3b8aa9" instance_seqno: 1781920565276409) as {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:56719; Asking this server to re-register.
I20260620 01:56:06.367826 13794 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:06.370170 13840 ts_manager.cc:194] Registered new tserver with Master: 917d6f4589cd4b4c9d9b30ac6d3b8aa9 (127.20.1.131:44667)
I20260620 01:56:06.581496 13651 heartbeater.cc:344] Connected to a master server at 127.20.1.190:45553
I20260620 01:56:06.581871 13651 heartbeater.cc:499] Master 127.20.1.190:45553 was elected leader, sending a full tablet report...
I20260620 01:56:06.583478 13840 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "733436d44f6343cdb1325bf4d4531311" instance_seqno: 1781920564479173) as {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:37951; Asking this server to re-register.
I20260620 01:56:06.584525 13651 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:06.586828 13840 ts_manager.cc:194] Registered new tserver with Master: 733436d44f6343cdb1325bf4d4531311 (127.20.1.130:46557)
W20260620 01:56:06.812790 13898 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:06.813284 13898 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:06.813855 13898 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:06.826505 13898 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:06.826758 13898 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:56:06.840365 13898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=34163
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:06.842747 13898 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:06.844892 13898 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:06.862174 13907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:06.862176 13906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:06.862373 13909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:06.863003 13898 server_base.cc:1061] running on GCE node
I20260620 01:56:06.863782 13898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:06.865342 13898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:06.866709 13898 hybrid_clock.cc:648] HybridClock initialized: now 1781920566866603 us; error 112 us; skew 500 ppm
I20260620 01:56:06.867378 13898 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:06 dist-test-slave-wg14 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920566, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:06.873369 13898 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:56:06.876451 13898 webserver.cc:533] Webserver started at http://127.20.1.129:34163/ using document root <none> and password file <none>
I20260620 01:56:06.877527 13898 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:06.877713 13898 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:06.886273 13898 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.003s
I20260620 01:56:06.890761 13916 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:06.892858 13898 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 01:56:06.893170 13898 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "317a641b28a547f9a076cf94c18c114a"
format_stamp: "Formatted at 2026-06-20 01:56:03 on dist-test-slave-wg14"
I20260620 01:56:06.894241 13898 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:06.913625 13898 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:06.919807 13898 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:06.920117 13898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:06.921463 13898 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:56:06.923842 13898 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:56:06.923976 13898 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:06.924096 13898 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:56:06.924172 13898 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:06.967731 13898 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35175
I20260620 01:56:06.967815 14028 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35175 every 8 connection(s)
I20260620 01:56:06.969787 13898 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:56:06.971225 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 13898
Jun 20 01:56:06 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Jun 20 01:56:06 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920566, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:56:07.020584 14040 raft_consensus.cc:493] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7400b9832ac44a1ad0dc8f8f101ea98)
Jun 20 01:56:07 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920566, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:56:07.020867 14040 raft_consensus.cc:515] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:07.021780 14029 heartbeater.cc:344] Connected to a master server at 127.20.1.190:45553
I20260620 01:56:07.022280 14029 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:07.023306 14040 leader_election.cc:290] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7400b9832ac44a1ad0dc8f8f101ea98 (127.20.1.190:45553), 1010357ab4514fdea43e3a6cf7d9ac9c (127.20.1.188:42201)
I20260620 01:56:07.023819 14029 heartbeater.cc:507] Master 127.20.1.190:45553 requested a full tablet report, sending...
I20260620 01:56:07.023999 14039 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7400b9832ac44a1ad0dc8f8f101ea98)
I20260620 01:56:07.024288 14039 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
Jun 20 01:56:07 dist-test-slave-wg14 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920566, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:56:07.026633 13277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5f2999892d543c2a5e963a26648f971" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" is_pre_election: true
I20260620 01:56:07.026777 14039 leader_election.cc:290] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7400b9832ac44a1ad0dc8f8f101ea98 (127.20.1.190:45553), d5f2999892d543c2a5e963a26648f971 (127.20.1.189:33725)
I20260620 01:56:07.026975 13277 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5f2999892d543c2a5e963a26648f971 in term 1.
I20260620 01:56:07.028457 13174 leader_election.cc:304] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1010357ab4514fdea43e3a6cf7d9ac9c, d5f2999892d543c2a5e963a26648f971; no voters:
I20260620 01:56:07.035660 14040 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:56:07.035739 14031 heartbeater.cc:344] Connected to a master server at 127.20.1.189:33725
I20260620 01:56:07.035965 14031 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:07.036062 14040 raft_consensus.cc:493] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f7400b9832ac44a1ad0dc8f8f101ea98)
I20260620 01:56:07.036223 14040 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:56:07.036765 14031 heartbeater.cc:507] Master 127.20.1.189:33725 requested a full tablet report, sending...
I20260620 01:56:07.039927 14040 raft_consensus.cc:515] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:07.041052 13188 ts_manager.cc:194] Re-registered known tserver with Master: 317a641b28a547f9a076cf94c18c114a (127.20.1.129:35175)
I20260620 01:56:07.042676 14040 leader_election.cc:290] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [CANDIDATE]: Term 2 election: Requested vote from peers f7400b9832ac44a1ad0dc8f8f101ea98 (127.20.1.190:45553), 1010357ab4514fdea43e3a6cf7d9ac9c (127.20.1.188:42201)
I20260620 01:56:07.043692 13198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d5f2999892d543c2a5e963a26648f971" is_pre_election: true
I20260620 01:56:07.044054 13198 raft_consensus.cc:2393] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1010357ab4514fdea43e3a6cf7d9ac9c in current term 2: Already voted for candidate d5f2999892d543c2a5e963a26648f971 in this term.
I20260620 01:56:07.045197 13277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5f2999892d543c2a5e963a26648f971" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c"
I20260620 01:56:07.045460 13277 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:56:07.049856 13277 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5f2999892d543c2a5e963a26648f971 in term 2.
I20260620 01:56:07.052855 13840 ts_manager.cc:194] Registered new tserver with Master: 317a641b28a547f9a076cf94c18c114a (127.20.1.129:35175)
I20260620 01:56:07.053532 14030 heartbeater.cc:344] Connected to a master server at 127.20.1.188:42201
I20260620 01:56:07.053725 14030 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:07.051470 13174 leader_election.cc:304] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1010357ab4514fdea43e3a6cf7d9ac9c, d5f2999892d543c2a5e963a26648f971; no voters:
I20260620 01:56:07.054145 14040 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 01:56:07.054342 14030 heartbeater.cc:507] Master 127.20.1.188:42201 requested a full tablet report, sending...
I20260620 01:56:07.054368 13850 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5f2999892d543c2a5e963a26648f971" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" is_pre_election: true
I20260620 01:56:07.055002 13850 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5f2999892d543c2a5e963a26648f971 in term 1.
I20260620 01:56:07.055809 13850 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5f2999892d543c2a5e963a26648f971" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98"
I20260620 01:56:07.056062 13850 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:56:07.056185 13267 ts_manager.cc:194] Re-registered known tserver with Master: 317a641b28a547f9a076cf94c18c114a (127.20.1.129:35175)
I20260620 01:56:07.059943 14040 raft_consensus.cc:697] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 2 LEADER]: Becoming Leader. State: Replica: d5f2999892d543c2a5e963a26648f971, State: Running, Role: LEADER
I20260620 01:56:07.060745 14040 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:07.063210 13850 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5f2999892d543c2a5e963a26648f971 in term 2.
I20260620 01:56:07.065073 13850 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" is_pre_election: true
I20260620 01:56:07.065537 14044 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5f2999892d543c2a5e963a26648f971. Latest consensus state: current_term: 2 leader_uuid: "d5f2999892d543c2a5e963a26648f971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:07.065997 13850 raft_consensus.cc:2393] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1010357ab4514fdea43e3a6cf7d9ac9c in current term 2: Already voted for candidate d5f2999892d543c2a5e963a26648f971 in this term.
I20260620 01:56:07.066246 14044 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: This master's current role is: LEADER
I20260620 01:56:07.067323 13253 leader_election.cc:304] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1010357ab4514fdea43e3a6cf7d9ac9c; no voters: d5f2999892d543c2a5e963a26648f971, f7400b9832ac44a1ad0dc8f8f101ea98
I20260620 01:56:07.067819 14039 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 01:56:07.068535 14046 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:56:07.071597 14046 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:56:07.072842 14046 catalog_manager.cc:1294] Loaded cluster ID: 582c0e6637124b4fad99bb90819d0134
I20260620 01:56:07.072988 14046 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:56:07.074896 14046 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:56:07.076665 14046 catalog_manager.cc:6080] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971: Loaded TSK: 0
I20260620 01:56:07.078166 14046 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:56:07.213508 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 13898
W20260620 01:56:07.215154 14050 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:56:07.206641 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.20.1.129:35175)
0620 01:56:07.209310 (+ 2669us) server_negotiation.cc:207] Beginning negotiation
0620 01:56:07.209322 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0620 01:56:07.210263 (+ 941us) server_negotiation.cc:408] Connection header received
0620 01:56:07.210685 (+ 422us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:56:07.210698 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:56:07.211272 (+ 574us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:56:07.211508 (+ 236us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:56:07.213387 (+ 1879us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37494: BlockingRecv error: recv got EOF from 127.0.0.1:37494 (error 108)
Metrics: {"server-negotiator.queue_time_us":2456,"thread_start_us":2189,"threads_started":1}
W20260620 01:56:07.216820 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 13898: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:35175: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:56:07.216970 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 13898
W20260620 01:56:07.216975 14050 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:56:07.214888 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.20.1.129:35175)
0620 01:56:07.215497 (+ 609us) server_negotiation.cc:207] Beginning negotiation
0620 01:56:07.215503 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0620 01:56:07.215526 (+ 23us) server_negotiation.cc:408] Connection header received
0620 01:56:07.215603 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:56:07.215609 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:56:07.215694 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:56:07.215818 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:56:07.216721 (+ 903us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37498: BlockingRecv error: recv got EOF from 127.0.0.1:37498 (error 108)
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":1406,"threads_started":1}
I20260620 01:56:07.234349 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35175
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--webserver_port=34163
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:56:07.493026 13850 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 2 FOLLOWER]: Refusing update from remote peer d5f2999892d543c2a5e963a26648f971: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:56:07.494557 14044 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:56:07.499696 14053 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5f2999892d543c2a5e963a26648f971. Latest consensus state: current_term: 2 leader_uuid: "d5f2999892d543c2a5e963a26648f971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:07.500113 14053 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:07.502260 14044 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d5f2999892d543c2a5e963a26648f971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:07.502570 14044 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: This master's current role is: LEADER
I20260620 01:56:07.503261 13277 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 2 FOLLOWER]: Refusing update from remote peer d5f2999892d543c2a5e963a26648f971: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:56:07.504343 14044 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:56:07.507472 14053 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d5f2999892d543c2a5e963a26648f971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:07.507764 14053 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:07.513298 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d5f2999892d543c2a5e963a26648f971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:07.513861 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:07.513726 14039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5f2999892d543c2a5e963a26648f971. Latest consensus state: current_term: 2 leader_uuid: "d5f2999892d543c2a5e963a26648f971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:07.514003 14039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:56:07.678822 14052 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:07.679329 14052 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:07.679509 14052 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:56:07.679584 14052 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:07.689994 14052 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:07.690287 14052 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:56:07.703763 14052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=34163
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:07.706086 14052 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:07.707864 14052 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:07.723322 14066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:07.723322 14067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:07.723873 14069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:07.726438 14052 server_base.cc:1061] running on GCE node
I20260620 01:56:07.727736 14052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:07.729207 14052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:07.730715 14052 hybrid_clock.cc:648] HybridClock initialized: now 1781920567730611 us; error 73 us; skew 500 ppm
I20260620 01:56:07.731317 14052 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:07 dist-test-slave-wg14 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920567, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:07.736581 14052 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:56:07.739701 14052 webserver.cc:533] Webserver started at http://127.20.1.129:34163/ using document root <none> and password file <none>
I20260620 01:56:07.740765 14052 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:07.740898 14052 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:07.749280 14052 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.002s
I20260620 01:56:07.754801 14076 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:07.757349 14052 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:56:07.757642 14052 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "317a641b28a547f9a076cf94c18c114a"
format_stamp: "Formatted at 2026-06-20 01:56:03 on dist-test-slave-wg14"
I20260620 01:56:07.758569 14052 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:07.782616 14052 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:07.788939 14052 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:07.789276 14052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:07.790735 14052 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:56:07.792927 14052 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:56:07.793052 14052 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:07.793195 14052 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:56:07.793252 14052 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:56:07.793901 14052 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:56:07.794004 14052 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:56:07.794108 14052 fault_injection.cc:59] injecting fault for kudu (pid 14052): FLAGS_fault_before_start (process will exit)
I20260620 01:56:07.810648 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35175
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--webserver_port=34163
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:56:07.972095 14040 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d5f2999892d543c2a5e963a26648f971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:07.972492 14040 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: This master's current role is: LEADER
W20260620 01:56:08.276412 14084 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:08.276893 14084 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:08.277034 14084 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:08.289511 14084 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:08.289816 14084 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:56:08.304514 14084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=34163
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:08.307680 14084 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:08.309485 14084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:08.325383 14089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:08.325246 14090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:08.326418 14092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:08.327160 14084 server_base.cc:1061] running on GCE node
I20260620 01:56:08.327912 14084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:08.329149 14084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:08.330461 14084 hybrid_clock.cc:648] HybridClock initialized: now 1781920568330398 us; error 61 us; skew 500 ppm
I20260620 01:56:08.330932 14084 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:08 dist-test-slave-wg14 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920568, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:08.335978 14084 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:56:08.338430 14084 webserver.cc:533] Webserver started at http://127.20.1.129:34163/ using document root <none> and password file <none>
I20260620 01:56:08.339686 14084 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:08.339896 14084 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:08.348816 14084 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:56:08.351828 13796 heartbeater.cc:499] Master 127.20.1.189:33725 was elected leader, sending a full tablet report...
I20260620 01:56:08.353060 14099 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:08.355696 14084 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260620 01:56:08.356222 14084 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "317a641b28a547f9a076cf94c18c114a"
format_stamp: "Formatted at 2026-06-20 01:56:03 on dist-test-slave-wg14"
I20260620 01:56:08.357800 14084 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:08.382059 14084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:08.387925 14084 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:08.388331 14084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:08.389863 14084 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:56:08.392428 14084 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:56:08.392582 14084 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:08.392719 14084 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:56:08.392797 14084 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:08.442454 14084 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35175
I20260620 01:56:08.442569 14211 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35175 every 8 connection(s)
I20260620 01:56:08.444545 14084 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:56:08.449522 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 14084
I20260620 01:56:08.449958 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 14084
W20260620 01:56:08.488914 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 14084: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:35175: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:56:08.489493 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 14084
I20260620 01:56:08.506717 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 13520
W20260620 01:56:08.507992 14218 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:56:08.453748 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:47053 (local address 127.20.1.189:33725)
0620 01:56:08.473811 (+ 20063us) server_negotiation.cc:207] Beginning negotiation
0620 01:56:08.473819 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:56:08.474382 (+ 563us) server_negotiation.cc:408] Connection header received
0620 01:56:08.478398 (+ 4016us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:56:08.478408 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:56:08.478594 (+ 186us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:56:08.478834 (+ 240us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:56:08.484681 (+ 5847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:56:08.487942 (+ 3261us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:56:08.506579 (+ 18637us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:47053: BlockingRecv error: recv got EOF from 127.20.1.129:47053 (error 108)
Metrics: {"server-negotiator.queue_time_us":19877,"thread_start_us":26164,"threads_started":1}
W20260620 01:56:08.507992 14220 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:56:08.460832 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:43339 (local address 127.20.1.188:42201)
0620 01:56:08.479896 (+ 19064us) server_negotiation.cc:207] Beginning negotiation
0620 01:56:08.479903 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 01:56:08.481337 (+ 1434us) server_negotiation.cc:408] Connection header received
0620 01:56:08.482120 (+ 783us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:56:08.482127 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:56:08.482225 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:56:08.482351 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:56:08.488925 (+ 6574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:56:08.493866 (+ 4941us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:56:08.506572 (+ 12706us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:43339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":18871,"thread_start_us":18217,"threads_started":1}
W20260620 01:56:08.507985 14222 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:56:08.475329 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:35753 (local address 127.20.1.190:45553)
0620 01:56:08.479871 (+ 4542us) server_negotiation.cc:207] Beginning negotiation
0620 01:56:08.479877 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0620 01:56:08.482012 (+ 2135us) server_negotiation.cc:408] Connection header received
0620 01:56:08.482259 (+ 247us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:56:08.482265 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:56:08.482368 (+ 103us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:56:08.482612 (+ 244us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:56:08.485594 (+ 2982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:56:08.488610 (+ 3016us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:56:08.506464 (+ 17854us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:35753: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":4404,"thread_start_us":4266,"threads_started":1}
W20260620 01:56:08.512032 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 13520: Not authorized: Client connection negotiation failed: client connection to 127.20.1.130:46557: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:56:08.512279 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 13520
I20260620 01:56:08.531186 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 13665
W20260620 01:56:08.535036 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 13665: Not authorized: Client connection negotiation failed: client connection to 127.20.1.131:44667: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:56:08.535255 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 13665
I20260620 01:56:08.553334 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 13806
W20260620 01:56:08.556866 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 13806: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:45553: server requires authentication, but client does not have Kerberos credentials available
W20260620 01:56:08.557065 14222 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:56:08.554988 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.20.1.190:45553)
0620 01:56:08.555368 (+ 380us) server_negotiation.cc:207] Beginning negotiation
0620 01:56:08.555375 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 01:56:08.555397 (+ 22us) server_negotiation.cc:408] Connection header received
0620 01:56:08.555478 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:56:08.555484 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:56:08.555577 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:56:08.555683 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:56:08.556704 (+ 1021us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45566: BlockingRecv error: recv got EOF from 127.0.0.1:45566 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 01:56:08.557163 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 13806
I20260620 01:56:08.580282 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 13156
I20260620 01:56:08.582135 13218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:56:08.717530 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 13156
I20260620 01:56:08.740516 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 13233
I20260620 01:56:08.742072 13298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:56:08.879504 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 13233
Shutting down hive metastore.
2026-06-20T01:56:09Z chronyd exiting
Jun 20 01:56:09 dist-test-slave-wg14 krb5kdc[14234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 01:56:09 dist-test-slave-wg14 krb5kdc[14234](info): set up 2 sockets
Jun 20 01:56:09 dist-test-slave-wg14 krb5kdc[14234](info): commencing operation
krb5kdc: starting...
W20260620 01:56:11.351670 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.039s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 01:56:11 dist-test-slave-wg14 krb5kdc[14234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T01:56:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:56:11Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "hive/127.0.0.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf
2026-06-20 01:56:12: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Jun 20 01:56:15 dist-test-slave-wg14 krb5kdc[14234](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920575, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 01:56:21 dist-test-slave-wg14 krb5kdc[14234](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920581, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 01:56:21 dist-test-slave-wg14 krb5kdc[14234](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920581, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.190@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.190@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:56:23.379735 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:45553
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:56:23.883529 14436 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:23.883912 14436 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:23.884009 14436 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:23.894695 14436 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:56:23.894855 14436 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:23.894917 14436 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:56:23.894960 14436 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:56:23.910861 14436 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:45553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:23.913471 14436 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:23.915364 14436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:23.927266 14442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:23.927417 14441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:23.927914 14444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:23.929339 14436 server_base.cc:1061] running on GCE node
I20260620 01:56:23.930639 14436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:23.933283 14436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:23.934679 14436 hybrid_clock.cc:648] HybridClock initialized: now 1781920583934588 us; error 60 us; skew 500 ppm
I20260620 01:56:23.935384 14436 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:23 dist-test-slave-wg14 krb5kdc[14234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920583, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:23.940456 14436 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:56:23.942778 14436 webserver.cc:533] Webserver started at http://127.20.1.190:38297/ using document root <none> and password file <none>
I20260620 01:56:23.943889 14436 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:23.944046 14436 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:23.951735 14436 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 01:56:23.957816 14451 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:23.960651 14436 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:56:23.961091 14436 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "f7400b9832ac44a1ad0dc8f8f101ea98"
format_stamp: "Formatted at 2026-06-20 01:56:00 on dist-test-slave-wg14"
I20260620 01:56:23.962224 14436 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:23.994426 14436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:24.004671 14436 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:24.004983 14436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:24.043839 14436 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:45553
I20260620 01:56:24.043867 14502 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:45553 every 8 connection(s)
I20260620 01:56:24.046074 14436 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:56:24.051041 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 14436
I20260620 01:56:24.052139 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:56:24.061025 14503 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:56:24.065685 14503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Bootstrap starting.
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.189@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:56:24.100703 14503 log.cc:826] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Log is configured to *not* fsync() on all Append() calls
I20260620 01:56:24.124881 14503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:56:24.125568 14503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Bootstrap complete.
I20260620 01:56:24.143785 14503 raft_consensus.cc:359] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:24.144888 14503 raft_consensus.cc:740] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7400b9832ac44a1ad0dc8f8f101ea98, State: Initialized, Role: FOLLOWER
I20260620 01:56:24.145939 14503 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:24.147877 14508 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:24.148778 14508 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:24.148944 14503 sys_catalog.cc:565] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: configured and running, proceeding with master startup.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:56:24.166639 14519 catalog_manager.cc:1294] Loaded cluster ID: 582c0e6637124b4fad99bb90819d0134
I20260620 01:56:24.167310 14519 catalog_manager.cc:1587] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: loading cluster ID for follower catalog manager: success
I20260620 01:56:24.179991 14519 catalog_manager.cc:1609] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: acquiring CA information for follower catalog manager: success
I20260620 01:56:24.184072 14519 catalog_manager.cc:1637] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.189@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:56:24.215728 14436 master_runner.cc:187] Error getting master registration for 127.20.1.189:33725: Network error: Client connection negotiation failed: client connection to 127.20.1.189:33725: connect: Connection refused (error 111),
I20260620 01:56:24.222430 14436 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:56:24.278620 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.189: real 0.125s user 0.000s sys 0.030s
I20260620 01:56:24.279652 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:33725
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:56:24.795243 14525 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:24.795640 14525 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:24.795755 14525 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:24.805821 14525 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:56:24.805958 14525 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:24.806018 14525 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:56:24.806087 14525 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:56:24.818212 14525 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:33725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:24.820843 14525 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:24.823171 14525 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:24.835379 14530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:24.836081 14531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:24.836019 14533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:24.837355 14525 server_base.cc:1061] running on GCE node
I20260620 01:56:24.838586 14525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:24.840546 14525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:24.841882 14525 hybrid_clock.cc:648] HybridClock initialized: now 1781920584841788 us; error 82 us; skew 500 ppm
I20260620 01:56:24.842446 14525 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:24 dist-test-slave-wg14 krb5kdc[14234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920584, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:24.847245 14525 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 01:56:24.849502 14525 webserver.cc:533] Webserver started at http://127.20.1.189:37853/ using document root <none> and password file <none>
I20260620 01:56:24.850528 14525 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:24.850708 14525 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:24.858062 14525 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:56:24.862012 14540 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:24.864233 14525 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 01:56:24.864543 14525 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "d5f2999892d543c2a5e963a26648f971"
format_stamp: "Formatted at 2026-06-20 01:56:01 on dist-test-slave-wg14"
I20260620 01:56:24.865590 14525 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:24.894887 14525 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:24.901683 14525 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:24.902066 14525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:24.926672 14525 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:33725
I20260620 01:56:24.926673 14592 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:33725 every 8 connection(s)
I20260620 01:56:24.928982 14525 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:56:24.938704 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 14525
I20260620 01:56:24.938964 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:56:24.939386 14593 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:56:24.944700 14593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971: Bootstrap starting.
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.188@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:56:24.971978 14593 log.cc:826] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971: Log is configured to *not* fsync() on all Append() calls
I20260620 01:56:24.988762 14593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:56:24.990217 14593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971: Bootstrap complete.
I20260620 01:56:25.000666 14593 raft_consensus.cc:359] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:25.001678 14593 raft_consensus.cc:740] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f2999892d543c2a5e963a26648f971, State: Initialized, Role: FOLLOWER
I20260620 01:56:25.002599 14593 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:25.003779 14598 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:25.004073 14598 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:25.005051 14593 sys_catalog.cc:565] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: configured and running, proceeding with master startup.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:56:25.025153 14609 catalog_manager.cc:1294] Loaded cluster ID: 582c0e6637124b4fad99bb90819d0134
I20260620 01:56:25.027535 14609 catalog_manager.cc:1587] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971: loading cluster ID for follower catalog manager: success
I20260620 01:56:25.032729 14609 catalog_manager.cc:1609] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971: acquiring CA information for follower catalog manager: success
I20260620 01:56:25.046399 14609 catalog_manager.cc:1637] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.188@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:56:25.079098 14525 master_runner.cc:187] Error getting master registration for 127.20.1.188:42201: Network error: Client connection negotiation failed: client connection to 127.20.1.188:42201: connect: Connection refused (error 111),
I20260620 01:56:25.083700 14525 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:56:25.108861 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:42201
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:56:25.629921 14616 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:25.630393 14616 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:25.630544 14616 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:25.640887 14616 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:56:25.641067 14616 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:25.641156 14616 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:56:25.641235 14616 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:56:25.655622 14616 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:42201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:25.657656 14616 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:25.659575 14616 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:25.671425 14622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:25.672082 14621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:25.672962 14624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:25.673642 14616 server_base.cc:1061] running on GCE node
I20260620 01:56:25.674711 14616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:25.677168 14616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:25.678498 14616 hybrid_clock.cc:648] HybridClock initialized: now 1781920585678411 us; error 92 us; skew 500 ppm
I20260620 01:56:25.679108 14616 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:25 dist-test-slave-wg14 krb5kdc[14234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920585, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:25.685370 14616 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 01:56:25.688608 14616 webserver.cc:533] Webserver started at http://127.20.1.188:39575/ using document root <none> and password file <none>
I20260620 01:56:25.689788 14616 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:25.689966 14616 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:25.699301 14616 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:56:25.703923 14631 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:25.706034 14616 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:56:25.706275 14616 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "1010357ab4514fdea43e3a6cf7d9ac9c"
format_stamp: "Formatted at 2026-06-20 01:56:02 on dist-test-slave-wg14"
I20260620 01:56:25.707366 14616 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:25.727727 14616 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:25.734342 14616 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:25.734697 14616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:25.761276 14616 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:42201
I20260620 01:56:25.761346 14682 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:42201 every 8 connection(s)
I20260620 01:56:25.763571 14616 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:56:25.765671 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 14616
I20260620 01:56:25.765926 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:56:25.775449 14683 sys_catalog.cc:263] Verifying existing consensus state
Jun 20 01:56:25 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:56:25.781417 14683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c: Bootstrap starting.
Jun 20 01:56:25 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:56:25.804684 14683 log.cc:826] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c: Log is configured to *not* fsync() on all Append() calls
I20260620 01:56:25.820709 14683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:56:25.821519 14683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c: Bootstrap complete.
Jun 20 01:56:25 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:56:25.831364 14683 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:25.832576 14683 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1010357ab4514fdea43e3a6cf7d9ac9c, State: Initialized, Role: FOLLOWER
I20260620 01:56:25.833511 14683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:25.834657 14690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:25.835050 14690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:25.836659 14683 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:56:25.851522 14701 catalog_manager.cc:1294] Loaded cluster ID: 582c0e6637124b4fad99bb90819d0134
I20260620 01:56:25.851747 14701 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c: loading cluster ID for follower catalog manager: success
I20260620 01:56:25.856495 14701 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c: acquiring CA information for follower catalog manager: success
I20260620 01:56:25.860626 14701 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Jun 20 01:56:25 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920585, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:56:25.905889 14616 master_runner.cc:264] No leader master found from master 1010357ab4514fdea43e3a6cf7d9ac9c
I20260620 01:56:25.906064 14616 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:56:25.956171 14704 raft_consensus.cc:493] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:56:25.956596 14704 raft_consensus.cc:515] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:25.959560 14704 leader_election.cc:290] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d5f2999892d543c2a5e963a26648f971 (127.20.1.189:33725), 1010357ab4514fdea43e3a6cf7d9ac9c (127.20.1.188:42201)
I20260620 01:56:25.972287 14567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d5f2999892d543c2a5e963a26648f971" is_pre_election: true
I20260620 01:56:25.973088 14567 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7400b9832ac44a1ad0dc8f8f101ea98 in term 2.
I20260620 01:56:25.973379 14658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" is_pre_election: true
I20260620 01:56:25.974048 14453 leader_election.cc:304] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5f2999892d543c2a5e963a26648f971, f7400b9832ac44a1ad0dc8f8f101ea98; no voters:
I20260620 01:56:25.974186 14658 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7400b9832ac44a1ad0dc8f8f101ea98 in term 2.
I20260620 01:56:25.974687 14704 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 01:56:25.974893 14704 raft_consensus.cc:493] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:56:25.975011 14704 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:56:25.981506 14704 raft_consensus.cc:515] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:25.983172 14704 leader_election.cc:290] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [CANDIDATE]: Term 3 election: Requested vote from peers d5f2999892d543c2a5e963a26648f971 (127.20.1.189:33725), 1010357ab4514fdea43e3a6cf7d9ac9c (127.20.1.188:42201)
I20260620 01:56:25.983330 14567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d5f2999892d543c2a5e963a26648f971"
I20260620 01:56:25.983618 14567 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:56:25.985275 14658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c"
I20260620 01:56:25.985581 14658 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:56:25.990196 14567 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7400b9832ac44a1ad0dc8f8f101ea98 in term 3.
I20260620 01:56:25.991048 14453 leader_election.cc:304] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5f2999892d543c2a5e963a26648f971, f7400b9832ac44a1ad0dc8f8f101ea98; no voters:
I20260620 01:56:25.991575 14704 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 01:56:25.992116 14658 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7400b9832ac44a1ad0dc8f8f101ea98 in term 3.
I20260620 01:56:25.992241 14704 raft_consensus.cc:697] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 3 LEADER]: Becoming Leader. State: Replica: f7400b9832ac44a1ad0dc8f8f101ea98, State: Running, Role: LEADER
I20260620 01:56:25.993319 14704 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:25.996547 14707 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7400b9832ac44a1ad0dc8f8f101ea98. Latest consensus state: current_term: 3 leader_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:25.997015 14707 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: This master's current role is: LEADER
I20260620 01:56:25.998127 14709 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:56:26.002461 14709 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:56:26.004354 14709 catalog_manager.cc:1294] Loaded cluster ID: 582c0e6637124b4fad99bb90819d0134
I20260620 01:56:26.004552 14709 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:56:26.006369 14709 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:56:26.007663 14709 catalog_manager.cc:6080] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Loaded TSK: 0
I20260620 01:56:26.009188 14709 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.129@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.129@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:56:26.182938 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:56:26.398458 14567 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 3 FOLLOWER]: Refusing update from remote peer f7400b9832ac44a1ad0dc8f8f101ea98: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:56:26.400046 14707 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:56:26.405607 14716 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7400b9832ac44a1ad0dc8f8f101ea98. Latest consensus state: current_term: 3 leader_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:26.406173 14716 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:26.407994 14707 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:26.408445 14707 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: This master's current role is: LEADER
I20260620 01:56:26.409812 14658 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 3 FOLLOWER]: Refusing update from remote peer f7400b9832ac44a1ad0dc8f8f101ea98: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:56:26.411162 14704 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:56:26.411339 14716 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:26.411540 14716 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:26.416129 14724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:26.416826 14724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:26.416872 14721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7400b9832ac44a1ad0dc8f8f101ea98. Latest consensus state: current_term: 3 leader_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:26.417131 14721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:56:26.674985 14715 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:26.675482 14715 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:26.675602 14715 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:26.688678 14715 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:26.688899 14715 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:56:26.703680 14715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:26.705613 14715 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:26.707901 14715 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:26.728297 14730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:26.728873 14729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:26.731081 14732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:26.731725 14715 server_base.cc:1061] running on GCE node
I20260620 01:56:26.732872 14715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:26.734201 14715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:26.736152 14715 hybrid_clock.cc:648] HybridClock initialized: now 1781920586735666 us; error 82 us; skew 500 ppm
I20260620 01:56:26.737162 14715 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:26 dist-test-slave-wg14 krb5kdc[14234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920586, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:26.743398 14715 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:56:26.745935 14715 webserver.cc:533] Webserver started at http://127.20.1.129:44387/ using document root <none> and password file <none>
I20260620 01:56:26.747320 14715 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:26.748030 14715 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:26.759644 14715 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 01:56:26.764180 14739 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:26.766337 14715 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:56:26.766600 14715 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "317a641b28a547f9a076cf94c18c114a"
format_stamp: "Formatted at 2026-06-20 01:56:03 on dist-test-slave-wg14"
I20260620 01:56:26.767603 14715 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:26.848920 14715 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:26.858855 14715 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:26.859266 14715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:26.860487 14715 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:56:26.863018 14715 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:56:26.863261 14715 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 01:56:26.863421 14715 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:56:26.863500 14715 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:26.910372 14704 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:26.910718 14704 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: This master's current role is: LEADER
I20260620 01:56:26.916846 14715 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:37737
I20260620 01:56:26.916898 14854 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:37737 every 8 connection(s)
I20260620 01:56:26.919580 14715 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:56:26.927887 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 14715
I20260620 01:56:26.928305 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 01:56:26 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920586, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:56:26 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920586, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:56:26.978924 14856 heartbeater.cc:344] Connected to a master server at 127.20.1.188:42201
I20260620 01:56:26.979535 14856 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:26.981360 14856 heartbeater.cc:507] Master 127.20.1.188:42201 requested a full tablet report, sending...
I20260620 01:56:26.984755 14648 ts_manager.cc:194] Registered new tserver with Master: 317a641b28a547f9a076cf94c18c114a (127.20.1.129:37737)
Jun 20 01:56:26 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920586, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:56:26.999202 14855 heartbeater.cc:344] Connected to a master server at 127.20.1.190:45553
I20260620 01:56:26.999516 14855 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:27.000221 14855 heartbeater.cc:507] Master 127.20.1.190:45553 requested a full tablet report, sending...
I20260620 01:56:27.002576 14468 ts_manager.cc:194] Registered new tserver with Master: 317a641b28a547f9a076cf94c18c114a (127.20.1.129:37737)
I20260620 01:56:27.005013 14468 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:33151
I20260620 01:56:27.011366 14857 heartbeater.cc:344] Connected to a master server at 127.20.1.189:33725
I20260620 01:56:27.011603 14857 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:27.012136 14857 heartbeater.cc:507] Master 127.20.1.189:33725 requested a full tablet report, sending...
I20260620 01:56:27.014658 14557 ts_manager.cc:194] Registered new tserver with Master: 317a641b28a547f9a076cf94c18c114a (127.20.1.129:37737)
WARNING: no policy specified for kudu/127.20.1.130@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.130@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.130 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.130 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:56:27.082705 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.130: real 0.137s user 0.001s sys 0.035s
WARNING: no policy specified for HTTP/127.20.1.130@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.130@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.130 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.130 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:56:27.169967 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:56:27.655262 14869 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:27.655762 14869 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:27.655920 14869 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:27.667001 14869 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:27.667452 14869 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:56:27.682993 14869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:27.685622 14869 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:27.687713 14869 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:27.704710 14874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:27.704679 14875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:27.705891 14877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:27.707077 14869 server_base.cc:1061] running on GCE node
I20260620 01:56:27.708916 14869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:27.711199 14869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:27.712996 14869 hybrid_clock.cc:648] HybridClock initialized: now 1781920587712950 us; error 139 us; skew 500 ppm
I20260620 01:56:27.714227 14869 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:27 dist-test-slave-wg14 krb5kdc[14234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920587, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:27.721256 14869 init.cc:377] Logged in from keytab as kudu/127.20.1.130@KRBTEST.COM (short username kudu)
I20260620 01:56:27.723652 14869 webserver.cc:533] Webserver started at http://127.20.1.130:40589/ using document root <none> and password file <none>
I20260620 01:56:27.724746 14869 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:27.725111 14869 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:27.739107 14869 fs_manager.cc:714] Time spent opening directory manager: real 0.009s user 0.009s sys 0.000s
I20260620 01:56:27.743816 14884 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:27.746937 14869 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260620 01:56:27.747328 14869 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "733436d44f6343cdb1325bf4d4531311"
format_stamp: "Formatted at 2026-06-20 01:56:04 on dist-test-slave-wg14"
I20260620 01:56:27.748792 14869 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:27.775846 14869 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:27.783495 14869 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:27.783824 14869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:27.785070 14869 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:56:27.787164 14869 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:56:27.787309 14869 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 01:56:27.787505 14869 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:56:27.787683 14869 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:27.877209 14869 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:37727
I20260620 01:56:27.877666 14996 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:37727 every 8 connection(s)
I20260620 01:56:27.881062 14869 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:56:27.885293 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 14869
I20260620 01:56:27.885735 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
Jun 20 01:56:27 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920587, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:56:27 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920587, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:56:27 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920587, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:56:27.968698 14999 heartbeater.cc:344] Connected to a master server at 127.20.1.189:33725
I20260620 01:56:27.968703 14998 heartbeater.cc:344] Connected to a master server at 127.20.1.188:42201
I20260620 01:56:27.969321 14999 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:27.969321 14998 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:27.971515 14999 heartbeater.cc:507] Master 127.20.1.189:33725 requested a full tablet report, sending...
I20260620 01:56:27.971608 14998 heartbeater.cc:507] Master 127.20.1.188:42201 requested a full tablet report, sending...
I20260620 01:56:27.974977 14557 ts_manager.cc:194] Registered new tserver with Master: 733436d44f6343cdb1325bf4d4531311 (127.20.1.130:37727)
I20260620 01:56:27.974974 14648 ts_manager.cc:194] Registered new tserver with Master: 733436d44f6343cdb1325bf4d4531311 (127.20.1.130:37727)
I20260620 01:56:27.976106 14997 heartbeater.cc:344] Connected to a master server at 127.20.1.190:45553
I20260620 01:56:27.976339 14997 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:27.977178 14997 heartbeater.cc:507] Master 127.20.1.190:45553 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.1.131@KRBTEST.COM; defaulting to no policy
I20260620 01:56:27.981206 14468 ts_manager.cc:194] Registered new tserver with Master: 733436d44f6343cdb1325bf4d4531311 (127.20.1.130:37727)
add_principal: Principal or policy already exists while creating "kudu/127.20.1.131@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:56:27.984319 14468 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:45761
I20260620 01:56:28.009781 14855 heartbeater.cc:499] Master 127.20.1.190:45553 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.131 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.131 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:56:28.049760 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.131: real 0.144s user 0.000s sys 0.033s
WARNING: no policy specified for HTTP/127.20.1.131@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.131@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.131 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.131 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:56:28.127497 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:56:28.601711 15011 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:28.602146 15011 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:28.602286 15011 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:28.613174 15011 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:28.613463 15011 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:56:28.627566 15011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:28.629494 15011 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:28.630998 15011 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:28.644815 15017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:28.645776 15020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:28.645793 15018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:28.646628 15011 server_base.cc:1061] running on GCE node
I20260620 01:56:28.647548 15011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:28.648820 15011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:28.650075 15011 hybrid_clock.cc:648] HybridClock initialized: now 1781920588650028 us; error 50 us; skew 500 ppm
I20260620 01:56:28.650564 15011 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:28 dist-test-slave-wg14 krb5kdc[14234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920588, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:28.655853 15011 init.cc:377] Logged in from keytab as kudu/127.20.1.131@KRBTEST.COM (short username kudu)
I20260620 01:56:28.658267 15011 webserver.cc:533] Webserver started at http://127.20.1.131:40589/ using document root <none> and password file <none>
I20260620 01:56:28.659348 15011 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:28.659528 15011 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:28.667210 15011 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:56:28.671751 15027 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:28.674201 15011 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.003s
I20260620 01:56:28.674499 15011 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "917d6f4589cd4b4c9d9b30ac6d3b8aa9"
format_stamp: "Formatted at 2026-06-20 01:56:05 on dist-test-slave-wg14"
I20260620 01:56:28.675417 15011 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:28.700670 15011 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:28.705999 15011 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:28.706352 15011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:28.707911 15011 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:56:28.710182 15011 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:56:28.710356 15011 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:28.710464 15011 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:56:28.710541 15011 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:28.759450 15011 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:35675
I20260620 01:56:28.759517 15140 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:35675 every 8 connection(s)
I20260620 01:56:28.761215 15011 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:56:28.768476 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 15011
I20260620 01:56:28.768769 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
Jun 20 01:56:28 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920588, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:56:28 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920588, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:56:28 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920588, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:56:28.829030 15141 heartbeater.cc:344] Connected to a master server at 127.20.1.190:45553
I20260620 01:56:28.829228 15143 heartbeater.cc:344] Connected to a master server at 127.20.1.189:33725
I20260620 01:56:28.829754 15143 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:28.829767 15141 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:28.832293 15141 heartbeater.cc:507] Master 127.20.1.190:45553 requested a full tablet report, sending...
I20260620 01:56:28.832297 15143 heartbeater.cc:507] Master 127.20.1.189:33725 requested a full tablet report, sending...
I20260620 01:56:28.832293 15142 heartbeater.cc:344] Connected to a master server at 127.20.1.188:42201
I20260620 01:56:28.832804 15142 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:28.833585 15142 heartbeater.cc:507] Master 127.20.1.188:42201 requested a full tablet report, sending...
I20260620 01:56:28.835552 14648 ts_manager.cc:194] Registered new tserver with Master: 917d6f4589cd4b4c9d9b30ac6d3b8aa9 (127.20.1.131:35675)
I20260620 01:56:28.835542 14557 ts_manager.cc:194] Registered new tserver with Master: 917d6f4589cd4b4c9d9b30ac6d3b8aa9 (127.20.1.131:35675)
I20260620 01:56:28.836354 14468 ts_manager.cc:194] Registered new tserver with Master: 917d6f4589cd4b4c9d9b30ac6d3b8aa9 (127.20.1.131:35675)
I20260620 01:56:28.837508 14468 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:48713
I20260620 01:56:28.852710 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:56:28.867163 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:56:28.867380 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 14436
I20260620 01:56:28.868925 14498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:56:28.993136 14997 heartbeater.cc:499] Master 127.20.1.190:45553 was elected leader, sending a full tablet report...
I20260620 01:56:29.014214 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 14436
W20260620 01:56:29.038528 14742 connection.cc:573] client connection to 127.20.1.190:45553 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 01:56:29.039104 14855 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:45553 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:56:29.039615 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:45553
--webserver_interface=127.20.1.190
--webserver_port=38297
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:56:29.520097 15152 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:29.520539 15152 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:29.520692 15152 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:29.532724 15152 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:56:29.532881 15152 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:29.532966 15152 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:56:29.533033 15152 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:56:29.547922 15152 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:45553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=38297
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:29.550047 15152 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:29.552337 15152 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:29.563730 15157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:29.563951 15160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:29.563796 15158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:29.564733 15152 server_base.cc:1061] running on GCE node
I20260620 01:56:29.565652 15152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:29.567982 15152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:29.569589 15152 hybrid_clock.cc:648] HybridClock initialized: now 1781920589569419 us; error 168 us; skew 500 ppm
I20260620 01:56:29.570087 15152 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:29 dist-test-slave-wg14 krb5kdc[14234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920589, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:29.575299 15152 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:56:29.578138 15152 webserver.cc:533] Webserver started at http://127.20.1.190:38297/ using document root <none> and password file <none>
I20260620 01:56:29.579820 15152 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:29.580264 15152 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:29.589003 15152 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:56:29.596251 15167 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:29.599267 15152 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.006s sys 0.002s
I20260620 01:56:29.599534 15152 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "f7400b9832ac44a1ad0dc8f8f101ea98"
format_stamp: "Formatted at 2026-06-20 01:56:00 on dist-test-slave-wg14"
I20260620 01:56:29.600548 15152 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:29.617304 15152 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:29.630211 15152 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:29.630564 15152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:29.668406 15152 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:45553
I20260620 01:56:29.668711 15219 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:45553 every 8 connection(s)
I20260620 01:56:29.670614 15152 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:56:29.673043 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 15152
I20260620 01:56:29.673441 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 14715
I20260620 01:56:29.682991 15220 sys_catalog.cc:263] Verifying existing consensus state
Jun 20 01:56:29 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 01:56:29.688957 15220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Bootstrap starting.
I20260620 01:56:29.706827 14849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:56:29.712009 15220 log.cc:826] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Log is configured to *not* fsync() on all Append() calls
I20260620 01:56:29.732414 15220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:56:29.733088 15220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: Bootstrap complete.
I20260620 01:56:29.741117 15220 raft_consensus.cc:359] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:29.741932 15220 raft_consensus.cc:740] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7400b9832ac44a1ad0dc8f8f101ea98, State: Initialized, Role: FOLLOWER
I20260620 01:56:29.742934 15220 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:29.744010 15226 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:29.744374 15226 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:29.745249 15220 sys_catalog.cc:565] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:56:29.774574 15237 catalog_manager.cc:1294] Loaded cluster ID: 582c0e6637124b4fad99bb90819d0134
I20260620 01:56:29.774847 15237 catalog_manager.cc:1587] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: loading cluster ID for follower catalog manager: success
I20260620 01:56:29.790037 15237 catalog_manager.cc:1609] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: acquiring CA information for follower catalog manager: success
I20260620 01:56:29.793799 15237 catalog_manager.cc:1637] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Jun 20 01:56:29 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920589, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:56:29.840759 15141 heartbeater.cc:499] Master 127.20.1.190:45553 was elected leader, sending a full tablet report...
I20260620 01:56:29.878075 15185 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "917d6f4589cd4b4c9d9b30ac6d3b8aa9" instance_seqno: 1781920588746846) as {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:40935; Asking this server to re-register.
I20260620 01:56:29.879645 15141 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:29.881897 15185 ts_manager.cc:194] Registered new tserver with Master: 917d6f4589cd4b4c9d9b30ac6d3b8aa9 (127.20.1.131:35675)
I20260620 01:56:29.906464 15152 master_runner.cc:264] No leader master found from master f7400b9832ac44a1ad0dc8f8f101ea98
I20260620 01:56:29.906677 15152 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:56:29.921363 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 14715
I20260620 01:56:29.943957 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:37737
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--webserver_port=44387
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:56:30.032884 15185 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "733436d44f6343cdb1325bf4d4531311" instance_seqno: 1781920587846541) as {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:39123; Asking this server to re-register.
I20260620 01:56:30.034596 14997 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:30.036953 15185 ts_manager.cc:194] Registered new tserver with Master: 733436d44f6343cdb1325bf4d4531311 (127.20.1.130:37727)
I20260620 01:56:30.124363 14525 master_runner.cc:264] No leader master found from master d5f2999892d543c2a5e963a26648f971
I20260620 01:56:30.124581 14525 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:56:30.204741 15247 raft_consensus.cc:493] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7400b9832ac44a1ad0dc8f8f101ea98)
I20260620 01:56:30.205204 15247 raft_consensus.cc:515] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:30.209141 15247 leader_election.cc:290] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7400b9832ac44a1ad0dc8f8f101ea98 (127.20.1.190:45553), 1010357ab4514fdea43e3a6cf7d9ac9c (127.20.1.188:42201)
I20260620 01:56:30.210141 14658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5f2999892d543c2a5e963a26648f971" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" is_pre_election: true
I20260620 01:56:30.211229 15195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5f2999892d543c2a5e963a26648f971" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" is_pre_election: true
I20260620 01:56:30.212117 15195 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5f2999892d543c2a5e963a26648f971 in term 3.
I20260620 01:56:30.213315 14543 leader_election.cc:304] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d5f2999892d543c2a5e963a26648f971, f7400b9832ac44a1ad0dc8f8f101ea98; no voters: 1010357ab4514fdea43e3a6cf7d9ac9c
I20260620 01:56:30.214344 15247 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 01:56:30.216138 15247 raft_consensus.cc:493] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f7400b9832ac44a1ad0dc8f8f101ea98)
I20260620 01:56:30.217152 15247 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 3 FOLLOWER]: Advancing to term 4
I20260620 01:56:30.221168 15247 raft_consensus.cc:515] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:30.222239 15247 leader_election.cc:290] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [CANDIDATE]: Term 4 election: Requested vote from peers f7400b9832ac44a1ad0dc8f8f101ea98 (127.20.1.190:45553), 1010357ab4514fdea43e3a6cf7d9ac9c (127.20.1.188:42201)
I20260620 01:56:30.223167 14658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5f2999892d543c2a5e963a26648f971" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c"
I20260620 01:56:30.223222 15195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5f2999892d543c2a5e963a26648f971" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98"
I20260620 01:56:30.223560 15195 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 3 FOLLOWER]: Advancing to term 4
I20260620 01:56:30.237394 15195 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5f2999892d543c2a5e963a26648f971 in term 4.
I20260620 01:56:30.238605 14543 leader_election.cc:304] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d5f2999892d543c2a5e963a26648f971, f7400b9832ac44a1ad0dc8f8f101ea98; no voters: 1010357ab4514fdea43e3a6cf7d9ac9c
I20260620 01:56:30.239324 15247 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 01:56:30.239985 15247 raft_consensus.cc:697] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [term 4 LEADER]: Becoming Leader. State: Replica: d5f2999892d543c2a5e963a26648f971, State: Running, Role: LEADER
I20260620 01:56:30.240566 15247 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:30.244145 15248 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5f2999892d543c2a5e963a26648f971. Latest consensus state: current_term: 4 leader_uuid: "d5f2999892d543c2a5e963a26648f971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:30.244638 15248 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: This master's current role is: LEADER
I20260620 01:56:30.245785 15250 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:56:30.250757 15250 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:56:30.252532 15250 catalog_manager.cc:1294] Loaded cluster ID: 582c0e6637124b4fad99bb90819d0134
I20260620 01:56:30.252697 15250 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:56:30.256966 15250 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:56:30.260480 15250 catalog_manager.cc:6080] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971: Loaded TSK: 0
I20260620 01:56:30.262277 15250 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:56:30.512931 15251 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7400b9832ac44a1ad0dc8f8f101ea98)
I20260620 01:56:30.513231 15251 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } }
I20260620 01:56:30.515776 15251 leader_election.cc:290] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7400b9832ac44a1ad0dc8f8f101ea98 (127.20.1.190:45553), d5f2999892d543c2a5e963a26648f971 (127.20.1.189:33725)
I20260620 01:56:30.517413 14567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d5f2999892d543c2a5e963a26648f971" is_pre_election: true
I20260620 01:56:30.527194 15195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" is_pre_election: true
I20260620 01:56:30.527716 15195 raft_consensus.cc:2393] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1010357ab4514fdea43e3a6cf7d9ac9c in current term 4: Already voted for candidate d5f2999892d543c2a5e963a26648f971 in this term.
I20260620 01:56:30.528888 14634 leader_election.cc:304] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1010357ab4514fdea43e3a6cf7d9ac9c; no voters: d5f2999892d543c2a5e963a26648f971, f7400b9832ac44a1ad0dc8f8f101ea98
I20260620 01:56:30.529410 15251 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 3 FOLLOWER]: Advancing to term 4
I20260620 01:56:30.532061 15251 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260620 01:56:30.578485 15244 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:30.579021 15244 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:30.579440 15244 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:30.593170 15244 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:30.593459 15244 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:56:30.610355 15244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:37737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=44387
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:30.613763 15244 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:30.615549 15244 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:30.634410 15257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:30.634508 15260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:30.634765 15258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:30.636729 15244 server_base.cc:1061] running on GCE node
I20260620 01:56:30.637666 15244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:30.639424 15244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:30.640950 15244 hybrid_clock.cc:648] HybridClock initialized: now 1781920590640846 us; error 74 us; skew 500 ppm
I20260620 01:56:30.641444 15244 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:30 dist-test-slave-wg14 krb5kdc[14234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920590, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:30.647799 15244 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:56:30.650768 15244 webserver.cc:533] Webserver started at http://127.20.1.129:44387/ using document root <none> and password file <none>
I20260620 01:56:30.651810 15244 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:30.651957 15244 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:30.659591 15244 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 01:56:30.665637 15267 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:30.668375 15244 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.004s
I20260620 01:56:30.668987 15244 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "317a641b28a547f9a076cf94c18c114a"
format_stamp: "Formatted at 2026-06-20 01:56:03 on dist-test-slave-wg14"
I20260620 01:56:30.672757 15244 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:30.697986 15244 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:30.703647 15244 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:30.703979 15244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:30.705214 15244 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:56:30.707278 15244 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:56:30.707509 15244 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:30.707765 15244 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:56:30.707854 15244 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:30.767556 15244 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:37737
I20260620 01:56:30.767896 15382 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:37737 every 8 connection(s)
I20260620 01:56:30.769811 15244 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:56:30.779914 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 15244
Jun 20 01:56:30 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
I20260620 01:56:30.819267 14658 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [term 4 FOLLOWER]: Refusing update from remote peer d5f2999892d543c2a5e963a26648f971: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260620 01:56:30.820432 15393 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
Jun 20 01:56:30 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920590, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:56:30 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920590, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:56:30.836989 15195 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [term 4 FOLLOWER]: Refusing update from remote peer d5f2999892d543c2a5e963a26648f971: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260620 01:56:30.838435 15393 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
Jun 20 01:56:30 dist-test-slave-wg14 krb5kdc[14234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920590, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:56:30.853117 15383 heartbeater.cc:344] Connected to a master server at 127.20.1.190:45553
I20260620 01:56:30.853718 15383 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:30.855182 15383 heartbeater.cc:507] Master 127.20.1.190:45553 requested a full tablet report, sending...
I20260620 01:56:30.859002 15251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5f2999892d543c2a5e963a26648f971. Latest consensus state: current_term: 4 leader_uuid: "d5f2999892d543c2a5e963a26648f971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:30.859387 15251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:30.867441 15385 heartbeater.cc:344] Connected to a master server at 127.20.1.189:33725
I20260620 01:56:30.868156 15385 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:30.868590 15251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "d5f2999892d543c2a5e963a26648f971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:30.869033 15251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1010357ab4514fdea43e3a6cf7d9ac9c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:30.869607 15385 heartbeater.cc:507] Master 127.20.1.189:33725 requested a full tablet report, sending...
I20260620 01:56:30.876587 15393 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "d5f2999892d543c2a5e963a26648f971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:30.876961 15393 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: This master's current role is: LEADER
I20260620 01:56:30.877542 14557 ts_manager.cc:194] Re-registered known tserver with Master: 317a641b28a547f9a076cf94c18c114a (127.20.1.129:37737)
I20260620 01:56:30.881603 15185 ts_manager.cc:194] Registered new tserver with Master: 317a641b28a547f9a076cf94c18c114a (127.20.1.129:37737)
I20260620 01:56:30.882777 14557 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:36975
I20260620 01:56:30.883983 15395 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5f2999892d543c2a5e963a26648f971. Latest consensus state: current_term: 4 leader_uuid: "d5f2999892d543c2a5e963a26648f971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:30.884397 15395 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:30.891712 15393 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "d5f2999892d543c2a5e963a26648f971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:30.892827 15393 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f2999892d543c2a5e963a26648f971 [sys.catalog]: This master's current role is: LEADER
I20260620 01:56:30.892716 15395 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "d5f2999892d543c2a5e963a26648f971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7400b9832ac44a1ad0dc8f8f101ea98" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 45553 } } peers { permanent_uuid: "d5f2999892d543c2a5e963a26648f971" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 33725 } } peers { permanent_uuid: "1010357ab4514fdea43e3a6cf7d9ac9c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 42201 } } }
I20260620 01:56:30.893045 15395 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7400b9832ac44a1ad0dc8f8f101ea98 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:56:30.898308 15384 heartbeater.cc:344] Connected to a master server at 127.20.1.188:42201
I20260620 01:56:30.898672 15384 heartbeater.cc:461] Registering TS with master...
I20260620 01:56:30.899443 15384 heartbeater.cc:507] Master 127.20.1.188:42201 requested a full tablet report, sending...
I20260620 01:56:30.901752 14648 ts_manager.cc:194] Re-registered known tserver with Master: 317a641b28a547f9a076cf94c18c114a (127.20.1.129:37737)
I20260620 01:56:31.041708 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 15244
W20260620 01:56:31.044880 15407 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:56:31.031889 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:58878 (local address 127.20.1.129:37737)
0620 01:56:31.033246 (+ 1357us) server_negotiation.cc:207] Beginning negotiation
0620 01:56:31.033258 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0620 01:56:31.033324 (+ 66us) server_negotiation.cc:408] Connection header received
0620 01:56:31.033427 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:56:31.033434 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:56:31.035205 (+ 1771us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:56:31.037583 (+ 2378us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:56:31.041676 (+ 4093us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58878: BlockingRecv error: recv got EOF from 127.0.0.1:58878 (error 108)
Metrics: {"server-negotiator.queue_time_us":985,"thread_start_us":543,"threads_started":1}
W20260620 01:56:31.046037 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 15244: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:37737: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:56:31.046262 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 15244
I20260620 01:56:31.079831 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:37737
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--webserver_port=44387
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:56:31.716607 15409 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:31.717168 15409 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:31.717334 15409 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:56:31.717406 15409 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:31.732697 15409 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:31.733002 15409 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:56:31.752218 15409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:37737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=44387
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:31.754328 15409 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:31.756162 15409 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:31.777050 15415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:31.778455 15417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:31.780778 15414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:31.781466 15409 server_base.cc:1061] running on GCE node
I20260620 01:56:31.782405 15409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:31.784214 15409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:31.786091 15409 hybrid_clock.cc:648] HybridClock initialized: now 1781920591785995 us; error 79 us; skew 500 ppm
I20260620 01:56:31.786527 15409 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:31 dist-test-slave-wg14 krb5kdc[14234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920591, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:31.800884 15409 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:56:31.803433 15409 webserver.cc:533] Webserver started at http://127.20.1.129:44387/ using document root <none> and password file <none>
I20260620 01:56:31.804469 15409 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:31.804615 15409 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:31.822837 15409 fs_manager.cc:714] Time spent opening directory manager: real 0.016s user 0.011s sys 0.007s
I20260620 01:56:31.827498 15425 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:31.831732 15409 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260620 01:56:31.833115 15409 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "317a641b28a547f9a076cf94c18c114a"
format_stamp: "Formatted at 2026-06-20 01:56:03 on dist-test-slave-wg14"
I20260620 01:56:31.835216 15409 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:31.850385 15143 heartbeater.cc:499] Master 127.20.1.189:33725 was elected leader, sending a full tablet report...
I20260620 01:56:31.865622 15409 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:31.876365 15409 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:31.876853 15409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:31.878261 15409 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:56:31.880559 15409 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:56:31.880692 15409 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:31.880825 15409 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:56:31.880892 15409 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:56:31.881371 15409 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:56:31.881466 15409 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:56:31.881541 15409 fault_injection.cc:59] injecting fault for kudu (pid 15409): FLAGS_fault_before_start (process will exit)
I20260620 01:56:31.906535 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:37737
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--webserver_port=44387
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:56:31.986646 14999 heartbeater.cc:499] Master 127.20.1.189:33725 was elected leader, sending a full tablet report...
W20260620 01:56:32.428675 15433 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:32.429159 15433 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:32.429307 15433 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:32.440618 15433 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:32.441042 15433 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:56:32.455816 15433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:37737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=44387
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:45553,127.20.1.189:33725,127.20.1.188:42201
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:32.457818 15433 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:32.459709 15433 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:32.474119 15438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:32.474730 15441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:32.474740 15439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:32.476899 15433 server_base.cc:1061] running on GCE node
I20260620 01:56:32.478209 15433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:32.479974 15433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:32.481565 15433 hybrid_clock.cc:648] HybridClock initialized: now 1781920592481462 us; error 73 us; skew 500 ppm
I20260620 01:56:32.482231 15433 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:32 dist-test-slave-wg14 krb5kdc[14234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920592, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:32.488036 15433 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:56:32.490916 15433 webserver.cc:533] Webserver started at http://127.20.1.129:44387/ using document root <none> and password file <none>
I20260620 01:56:32.491974 15433 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:32.492154 15433 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:32.499094 15433 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:56:32.503361 15448 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:32.505465 15433 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.003s
I20260620 01:56:32.505708 15433 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "317a641b28a547f9a076cf94c18c114a"
format_stamp: "Formatted at 2026-06-20 01:56:03 on dist-test-slave-wg14"
I20260620 01:56:32.506925 15433 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:32.537361 15433 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:32.543357 15433 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:32.543701 15433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:32.545059 15433 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:56:32.547232 15433 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:56:32.547389 15433 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:32.547773 15433 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:56:32.547909 15433 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:32.604394 15433 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:37737
I20260620 01:56:32.604470 15561 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:37737 every 8 connection(s)
I20260620 01:56:32.606323 15433 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:56:32.609020 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 15433
I20260620 01:56:32.609477 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 15433
W20260620 01:56:32.642026 15569 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:56:32.616922 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:32826 (local address 127.20.1.129:37737)
0620 01:56:32.635989 (+ 19067us) server_negotiation.cc:207] Beginning negotiation
0620 01:56:32.635998 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0620 01:56:32.636548 (+ 550us) server_negotiation.cc:408] Connection header received
0620 01:56:32.636971 (+ 423us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:56:32.636986 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:56:32.638085 (+ 1099us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:56:32.638367 (+ 282us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:56:32.639821 (+ 1454us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32826: BlockingRecv error: recv got EOF from 127.0.0.1:32826 (error 108)
Metrics: {"server-negotiator.queue_time_us":18838,"thread_start_us":18520,"threads_started":1}
W20260620 01:56:32.643430 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 15433: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:37737: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:56:32.643716 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 15433
I20260620 01:56:32.659667 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 14869
W20260620 01:56:32.661080 15573 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:56:32.637998 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:47911 (local address 127.20.1.188:42201)
0620 01:56:32.645413 (+ 7415us) server_negotiation.cc:207] Beginning negotiation
0620 01:56:32.645418 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0620 01:56:32.645444 (+ 26us) server_negotiation.cc:408] Connection header received
0620 01:56:32.659463 (+ 14019us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:47911: BlockingRecv error: recv got EOF from 127.20.1.129:47911 (error 108)
Metrics: {"server-negotiator.queue_time_us":7209,"thread_start_us":8596,"threads_started":1}
W20260620 01:56:32.661080 15568 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:56:32.616463 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:38557 (local address 127.20.1.189:33725)
0620 01:56:32.639781 (+ 23318us) server_negotiation.cc:207] Beginning negotiation
0620 01:56:32.639790 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0620 01:56:32.639816 (+ 26us) server_negotiation.cc:408] Connection header received
0620 01:56:32.639893 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:56:32.639899 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:56:32.639987 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:56:32.640117 (+ 130us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:56:32.659463 (+ 19346us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:38557: BlockingRecv error: recv got EOF from 127.20.1.129:38557 (error 108)
Metrics: {"server-negotiator.queue_time_us":21665,"thread_start_us":21533,"threads_started":1}
W20260620 01:56:32.661247 15567 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:56:32.615161 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:60235 (local address 127.20.1.190:45553)
0620 01:56:32.640699 (+ 25538us) server_negotiation.cc:207] Beginning negotiation
0620 01:56:32.640706 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 01:56:32.640729 (+ 23us) server_negotiation.cc:408] Connection header received
0620 01:56:32.640811 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:56:32.640817 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:56:32.640932 (+ 115us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:56:32.641096 (+ 164us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:56:32.659550 (+ 18454us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:60235: BlockingRecv error: recv got EOF from 127.20.1.129:60235 (error 108)
Metrics: {"server-negotiator.queue_time_us":25386,"thread_start_us":25234,"threads_started":1}
W20260620 01:56:32.664916 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 14869: Not authorized: Client connection negotiation failed: client connection to 127.20.1.130:37727: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:56:32.665146 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 14869
I20260620 01:56:32.679536 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 15011
W20260620 01:56:32.685015 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 15011: Not authorized: Client connection negotiation failed: client connection to 127.20.1.131:35675: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:56:32.685221 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 15011
I20260620 01:56:32.702647 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 15152
W20260620 01:56:32.705905 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 15152: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:45553: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:56:32.706158 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 15152
W20260620 01:56:32.706215 15567 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:56:32.704110 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:58040 (local address 127.20.1.190:45553)
0620 01:56:32.704369 (+ 259us) server_negotiation.cc:207] Beginning negotiation
0620 01:56:32.704374 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0620 01:56:32.704425 (+ 51us) server_negotiation.cc:408] Connection header received
0620 01:56:32.704503 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:56:32.704508 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:56:32.704591 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:56:32.704682 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:56:32.705811 (+ 1129us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58040: BlockingRecv error: recv got EOF from 127.0.0.1:58040 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 01:56:32.729034 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 14525
I20260620 01:56:32.730806 14588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:56:32.871196 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 14525
I20260620 01:56:32.900214 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 14616
I20260620 01:56:32.902560 14678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:56:33.038640 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 14616
Shutting down hive metastore.
2026-06-20T01:56:33Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/10 (56166 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/11
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 01:56:33 dist-test-slave-wg14 krb5kdc[15585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 01:56:33 dist-test-slave-wg14 krb5kdc[15585](info): set up 2 sockets
Jun 20 01:56:33 dist-test-slave-wg14 krb5kdc[15585](info): commencing operation
krb5kdc: starting...
W20260620 01:56:35.804714 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.319s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 01:56:36 dist-test-slave-wg14 krb5kdc[15585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T01:56:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:56:36Z Disabled control of system clock
2026-06-20T01:56:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:56:36Z Disabled control of system clock
2026-06-20T01:56:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:56:36Z Disabled control of system clock
2026-06-20T01:56:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:56:36Z Disabled control of system clock
2026-06-20T01:56:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:56:36Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf
2026-06-20 01:56:47: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Jun 20 01:56:49 dist-test-slave-wg14 krb5kdc[15585](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920609, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 01:56:57 dist-test-slave-wg14 krb5kdc[15585](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920617, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 01:56:57 dist-test-slave-wg14 krb5kdc[15585](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920617, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:56:57.940748 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:35471
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:56:58.394097 15894 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:58.394563 15894 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:58.394682 15894 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:58.404934 15894 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:56:58.405084 15894 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:58.405144 15894 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:56:58.405184 15894 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:56:58.418260 15894 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:35471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:58.420457 15894 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:58.422384 15894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:58.434664 15900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:58.434762 15899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:58.434809 15902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:58.435672 15894 server_base.cc:1061] running on GCE node
I20260620 01:56:58.436785 15894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:58.439651 15894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:58.443281 15894 hybrid_clock.cc:648] HybridClock initialized: now 1781920618443004 us; error 191 us; skew 500 ppm
I20260620 01:56:58.443908 15894 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:58 dist-test-slave-wg14 krb5kdc[15585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920618, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:58.449203 15894 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:56:58.452013 15894 webserver.cc:533] Webserver started at http://127.20.1.190:34379/ using document root <none> and password file <none>
I20260620 01:56:58.453099 15894 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:58.453274 15894 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:58.453725 15894 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:56:58.457304 15894 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "5e1a5c80089949c18d5b0336c188a008"
format_stamp: "Formatted at 2026-06-20 01:56:58 on dist-test-slave-wg14"
I20260620 01:56:58.458457 15894 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "5e1a5c80089949c18d5b0336c188a008"
format_stamp: "Formatted at 2026-06-20 01:56:58 on dist-test-slave-wg14"
I20260620 01:56:58.466233 15894 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260620 01:56:58.473069 15909 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:58.476364 15894 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260620 01:56:58.476702 15894 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "5e1a5c80089949c18d5b0336c188a008"
format_stamp: "Formatted at 2026-06-20 01:56:58 on dist-test-slave-wg14"
I20260620 01:56:58.477012 15894 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:58.511796 15894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:58.522336 15894 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:58.522758 15894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:58.554893 15894 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:35471
I20260620 01:56:58.554939 15961 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:35471 every 8 connection(s)
I20260620 01:56:58.556787 15894 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:56:58.562472 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 15894
I20260620 01:56:58.562996 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:56:58.564801 15962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:56:58.574404 15962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:56:58 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920618, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.189@KRBTEST.COM" created.
I20260620 01:56:58.610558 15962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:56:58.612709 15911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:37567: connect: Connection refused (error 111)
W20260620 01:56:58.615319 15962 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:37567: Network error: Client connection negotiation failed: client connection to 127.20.1.189:37567: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:56:58.664848 15962 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } attempt: 1
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
W20260620 01:56:58.668020 15962 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:37567: Network error: Client connection negotiation failed: client connection to 127.20.1.189:37567: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:56:58.708173 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:37567
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:56:58.736388 15962 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } attempt: 2
W20260620 01:56:58.739774 15962 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:37567: Network error: Client connection negotiation failed: client connection to 127.20.1.189:37567: connect: Connection refused (error 111)
I20260620 01:56:58.831247 15962 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } attempt: 3
W20260620 01:56:58.834370 15962 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:37567: Network error: Client connection negotiation failed: client connection to 127.20.1.189:37567: connect: Connection refused (error 111)
I20260620 01:56:58.977769 15962 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } attempt: 4
W20260620 01:56:58.981112 15962 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:37567: Network error: Client connection negotiation failed: client connection to 127.20.1.189:37567: connect: Connection refused (error 111)
W20260620 01:56:59.149410 15971 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:59.149853 15971 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:59.149994 15971 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:59.161101 15971 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:56:59.161527 15971 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:59.161731 15971 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:56:59.161809 15971 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:56:59.175194 15971 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:37567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:59.177184 15971 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:59.178918 15971 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:59.192992 15976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:59.193137 15977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:59.195613 15979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:59.196372 15971 server_base.cc:1061] running on GCE node
I20260620 01:56:59.197371 15971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:59.199556 15971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:56:59.201176 15971 hybrid_clock.cc:648] HybridClock initialized: now 1781920619200986 us; error 112 us; skew 500 ppm
I20260620 01:56:59.201783 15971 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:59 dist-test-slave-wg14 krb5kdc[15585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920619, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:59.207557 15971 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 01:56:59.210517 15971 webserver.cc:533] Webserver started at http://127.20.1.189:40831/ using document root <none> and password file <none>
I20260620 01:56:59.211611 15971 fs_manager.cc:362] Metadata directory not provided
I20260620 01:56:59.211759 15971 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:56:59.212121 15971 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:56:59.215289 15971 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "7a004ebbc3c54fbe98351e428a62af87"
format_stamp: "Formatted at 2026-06-20 01:56:59 on dist-test-slave-wg14"
I20260620 01:56:59.216506 15971 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "7a004ebbc3c54fbe98351e428a62af87"
format_stamp: "Formatted at 2026-06-20 01:56:59 on dist-test-slave-wg14"
I20260620 01:56:59.225708 15971 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260620 01:56:59.231549 15986 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:56:59.234231 15971 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260620 01:56:59.234488 15971 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "7a004ebbc3c54fbe98351e428a62af87"
format_stamp: "Formatted at 2026-06-20 01:56:59 on dist-test-slave-wg14"
I20260620 01:56:59.234750 15971 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:56:59.258260 15971 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:56:59.264716 15971 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:56:59.265110 15971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:56:59.280534 15962 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } attempt: 5
W20260620 01:56:59.283635 15962 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:37567: Network error: Client connection negotiation failed: client connection to 127.20.1.189:37567: connect: Connection refused (error 111)
I20260620 01:56:59.293849 15971 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:37567
I20260620 01:56:59.293849 16037 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:37567 every 8 connection(s)
I20260620 01:56:59.296056 15971 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:56:59.301919 16038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:56:59.305164 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 15971
I20260620 01:56:59.305429 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:56:59.311949 16038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:56:59 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920619, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.188@KRBTEST.COM" created.
I20260620 01:56:59.350965 16038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:56:59 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920619, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:56:59.376634 16038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:56:59.379344 15990 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37811: connect: Connection refused (error 111)
W20260620 01:56:59.382622 16038 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37811: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37811: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.188@KRBTEST.COM" created.
I20260620 01:56:59.432214 16038 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } attempt: 1
W20260620 01:56:59.434458 16038 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37811: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37811: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:56:59.464192 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:37811
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:56:59.502916 16038 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } attempt: 2
W20260620 01:56:59.506680 16038 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37811: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37811: connect: Connection refused (error 111)
I20260620 01:56:59.598057 16038 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } attempt: 3
W20260620 01:56:59.601082 16038 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37811: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37811: connect: Connection refused (error 111)
I20260620 01:56:59.744462 16038 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } attempt: 4
W20260620 01:56:59.747439 16038 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37811: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37811: connect: Connection refused (error 111)
I20260620 01:56:59.831051 15962 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } attempt: 6
Jun 20 01:56:59 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920618, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:56:59.852944 15962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:56:59.856899 15962 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37811: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37811: connect: Connection refused (error 111)
I20260620 01:56:59.909248 15962 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } attempt: 1
W20260620 01:56:59.911653 15962 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37811: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37811: connect: Connection refused (error 111)
W20260620 01:56:59.938670 16048 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:56:59.939101 16048 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:56:59.939270 16048 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:56:59.949986 16048 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:56:59.950135 16048 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:56:59.950196 16048 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:56:59.950238 16048 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:56:59.965792 16048 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:37811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:56:59.968111 16048 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:56:59.969972 16048 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:56:59.983292 16056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:59.983161 16058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:56:59.983150 16055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:56:59.985263 16048 server_base.cc:1061] running on GCE node
I20260620 01:56:59.986233 15962 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } attempt: 2
I20260620 01:56:59.986512 16048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:56:59.989485 16048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260620 01:56:59.989745 15962 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37811: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37811: connect: Connection refused (error 111)
I20260620 01:56:59.991221 16048 hybrid_clock.cc:648] HybridClock initialized: now 1781920619990914 us; error 127 us; skew 500 ppm
I20260620 01:56:59.991861 16048 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:56:59 dist-test-slave-wg14 krb5kdc[15585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920619, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:56:59.998347 16048 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 01:57:00.001230 16048 webserver.cc:533] Webserver started at http://127.20.1.188:36747/ using document root <none> and password file <none>
I20260620 01:57:00.002336 16048 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:00.002533 16048 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:00.002956 16048 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:00.006692 16048 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "9b772652276745cabab57b0bd8ef5983"
format_stamp: "Formatted at 2026-06-20 01:56:59 on dist-test-slave-wg14"
I20260620 01:57:00.008095 16048 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "9b772652276745cabab57b0bd8ef5983"
format_stamp: "Formatted at 2026-06-20 01:56:59 on dist-test-slave-wg14"
I20260620 01:57:00.018121 16048 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260620 01:57:00.027043 16065 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:00.030121 16048 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260620 01:57:00.030443 16048 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "9b772652276745cabab57b0bd8ef5983"
format_stamp: "Formatted at 2026-06-20 01:56:59 on dist-test-slave-wg14"
I20260620 01:57:00.030722 16048 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:00.046888 16038 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } attempt: 5
I20260620 01:57:00.050590 16048 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20260620 01:57:00.050711 16038 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37811: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37811: connect: Connection refused (error 111)
I20260620 01:57:00.057050 16048 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:00.057431 16048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:00.092159 16048 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:37811
I20260620 01:57:00.092216 16117 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:37811 every 8 connection(s)
I20260620 01:57:00.094132 16048 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:57:00.095928 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 16048
I20260620 01:57:00.096174 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:57:00.102209 16118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:57:00.103011 15962 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } attempt: 3
Jun 20 01:57:00 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:57:00.115182 16118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:57:00 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920618, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:57:00 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:57:00 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920619, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:57:00.155259 15962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Bootstrap starting.
I20260620 01:57:00.160179 16118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:00.163044 15962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Neither blocks nor log segments found. Creating new log.
I20260620 01:57:00.165223 15962 log.cc:826] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Log is configured to *not* fsync() on all Append() calls
Jun 20 01:57:00 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:57:00.169818 15962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: No bootstrap required, opened a new log
Jun 20 01:57:00 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920619, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:57:00.181478 15962 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:00.182003 15962 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:57:00.182135 16118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:00.182181 15962 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1a5c80089949c18d5b0336c188a008, State: Initialized, Role: FOLLOWER
I20260620 01:57:00.183280 15962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:00.185415 16127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:00.185917 16127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:00.187178 15962 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 01:57:00 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920619, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:57:00.212889 16118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983: Bootstrap starting.
W20260620 01:57:00.217589 16138 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:57:00.217913 16138 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:57:00.222162 16118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983: Neither blocks nor log segments found. Creating new log.
I20260620 01:57:00.208359 15894 master_runner.cc:187] Error getting master registration for 127.20.1.189:37567: OK, instance_id { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" instance_seqno: 1781920619286546 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:57:00.223016 15894 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:57:00.224043 16118 log.cc:826] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:00.227833 16118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983: No bootstrap required, opened a new log
I20260620 01:57:00.240217 16118 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:00.240682 16118 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:57:00.240825 16118 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b772652276745cabab57b0bd8ef5983, State: Initialized, Role: FOLLOWER
I20260620 01:57:00.241757 16118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:00.244280 16141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:00.244948 16141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:00.245713 16118 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:00.245821 16127 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:57:00.247340 16127 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:00.251395 16092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e1a5c80089949c18d5b0336c188a008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b772652276745cabab57b0bd8ef5983" is_pre_election: true
I20260620 01:57:00.252408 16092 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1a5c80089949c18d5b0336c188a008 in term 0.
I20260620 01:57:00.254189 15913 leader_election.cc:304] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e1a5c80089949c18d5b0336c188a008, 9b772652276745cabab57b0bd8ef5983; no voters:
I20260620 01:57:00.255092 16127 leader_election.cc:290] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a004ebbc3c54fbe98351e428a62af87 (127.20.1.189:37567), 9b772652276745cabab57b0bd8ef5983 (127.20.1.188:37811)
I20260620 01:57:00.255723 16127 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:57:00.256021 16127 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:57:00.256155 16127 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:57:00.261859 16127 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:00.268232 16092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e1a5c80089949c18d5b0336c188a008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b772652276745cabab57b0bd8ef5983"
I20260620 01:57:00.268770 16092 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:57:00.276257 16092 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1a5c80089949c18d5b0336c188a008 in term 1.
I20260620 01:57:00.277091 15913 leader_election.cc:304] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e1a5c80089949c18d5b0336c188a008, 9b772652276745cabab57b0bd8ef5983; no voters:
I20260620 01:57:00.277489 16127 leader_election.cc:290] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [CANDIDATE]: Term 1 election: Requested vote from peers 7a004ebbc3c54fbe98351e428a62af87 (127.20.1.189:37567), 9b772652276745cabab57b0bd8ef5983 (127.20.1.188:37811)
I20260620 01:57:00.277786 16127 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 1 FOLLOWER]: Leader election won for term 1
W20260620 01:57:00.277876 15915 tablet.cc:2406] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:57:00.277964 16127 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1a5c80089949c18d5b0336c188a008, State: Running, Role: LEADER
I20260620 01:57:00.278762 16127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:00.302851 16148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e1a5c80089949c18d5b0336c188a008. Latest consensus state: current_term: 1 leader_uuid: "5e1a5c80089949c18d5b0336c188a008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:00.303651 16148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: This master's current role is: LEADER
W20260620 01:57:00.311195 16071 tablet.cc:2406] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:57:00.316891 16048 master_runner.cc:187] Error getting master registration for 127.20.1.189:37567: OK, instance_id { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" instance_seqno: 1781920619286546 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:57:00.332859 16048 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:57:00.338770 16155 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:57:00.342864 16155 catalog_manager.cc:1519] Initializing Kudu cluster ID...
W20260620 01:57:00.370126 16154 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:57:00.370332 16154 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:57:00.377785 16092 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 1 FOLLOWER]: Refusing update from remote peer 5e1a5c80089949c18d5b0336c188a008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:57:00.379453 16148 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:57:00.397665 16141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e1a5c80089949c18d5b0336c188a008. Latest consensus state: current_term: 1 leader_uuid: "5e1a5c80089949c18d5b0336c188a008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:00.398097 16141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:00.408571 16157 mvcc.cc:204] Tried to move back new op lower bound from 7298746861042245632 to 7298746860681916416. Current Snapshot: MvccSnapshot[applied={T|T < 7298746861042245632}]
I20260620 01:57:00.411221 16127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e1a5c80089949c18d5b0336c188a008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:00.411669 16127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:00.412676 16158 mvcc.cc:204] Tried to move back new op lower bound from 7298746861042245632 to 7298746860681916416. Current Snapshot: MvccSnapshot[applied={T|T < 7298746861042245632}]
I20260620 01:57:00.418079 16141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e1a5c80089949c18d5b0336c188a008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:00.418421 16141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:00.424556 16155 catalog_manager.cc:1382] Generated new cluster ID: c67ebe9c0bd849748e5f573512cf92ac
I20260620 01:57:00.424793 16155 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:57:00.476498 16155 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:57:00.478806 16155 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:57:00.515623 16155 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Generated new TSK 0
I20260620 01:57:00.517108 16155 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:57:00.598042 16038 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } attempt: 6
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Jun 20 01:57:00 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920619, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:57:00.675400 16038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87: Bootstrap starting.
I20260620 01:57:00.686940 16038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87: Neither blocks nor log segments found. Creating new log.
I20260620 01:57:00.690610 16038 log.cc:826] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:00.697252 16038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87: No bootstrap required, opened a new log
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:57:00.702910 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.150s user 0.000s sys 0.033s
I20260620 01:57:00.709242 16038 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:00.709836 16038 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:57:00.710019 16038 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a004ebbc3c54fbe98351e428a62af87, State: Initialized, Role: FOLLOWER
I20260620 01:57:00.711210 16038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:00.732336 16166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:00.732431 16011 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:57:00.732511 16013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e1a5c80089949c18d5b0336c188a008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a004ebbc3c54fbe98351e428a62af87" is_pre_election: true
I20260620 01:57:00.732488 16012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e1a5c80089949c18d5b0336c188a008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a004ebbc3c54fbe98351e428a62af87"
I20260620 01:57:00.738462 16011 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 1 FOLLOWER]: Refusing update from remote peer 5e1a5c80089949c18d5b0336c188a008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:57:00.732409 16038 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:00.740067 16148 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:57:00.741431 16166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:57:00.770361 15992 tablet.cc:2406] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:57:00.802343 16166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e1a5c80089949c18d5b0336c188a008. Latest consensus state: current_term: 1 leader_uuid: "5e1a5c80089949c18d5b0336c188a008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:00.802721 16166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:00.802973 16174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e1a5c80089949c18d5b0336c188a008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:00.803449 16174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: This master's current role is: FOLLOWER
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
I20260620 01:57:00.891814 16185 catalog_manager.cc:1294] Loaded cluster ID: c67ebe9c0bd849748e5f573512cf92ac
I20260620 01:57:00.892038 16185 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87: loading cluster ID for follower catalog manager: success
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:57:00.895356 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.192s user 0.004s sys 0.033s
I20260620 01:57:00.896517 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:57:00.910480 16185 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87: acquiring CA information for follower catalog manager: success
I20260620 01:57:00.914240 16185 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:57:01.264935 16148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e1a5c80089949c18d5b0336c188a008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:01.266250 16148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:01.374264 16154 catalog_manager.cc:1294] Loaded cluster ID: c67ebe9c0bd849748e5f573512cf92ac
I20260620 01:57:01.374408 16154 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983: loading cluster ID for follower catalog manager: success
I20260620 01:57:01.379717 16154 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983: acquiring CA information for follower catalog manager: success
I20260620 01:57:01.383540 16154 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 01:57:01.517668 16187 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:01.518095 16187 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:01.518236 16187 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:01.529947 16187 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:01.530257 16187 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:01.543668 16187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:01.546886 16187 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:01.549609 16187 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:01.564268 16195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:01.564438 16196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:01.566032 16198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:01.566574 16187 server_base.cc:1061] running on GCE node
I20260620 01:57:01.567428 16187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:01.568938 16187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:01.570477 16187 hybrid_clock.cc:648] HybridClock initialized: now 1781920621570257 us; error 167 us; skew 500 ppm
I20260620 01:57:01.570947 16187 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:57:01 dist-test-slave-wg14 krb5kdc[15585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920621, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:57:01.575788 16187 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:57:01.578583 16187 webserver.cc:533] Webserver started at http://127.20.1.129:35687/ using document root <none> and password file <none>
I20260620 01:57:01.579769 16187 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:01.579946 16187 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:01.580397 16187 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:01.583635 16187 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "240c1ed18e4b412f8997d62a51aa7b6a"
format_stamp: "Formatted at 2026-06-20 01:57:01 on dist-test-slave-wg14"
I20260620 01:57:01.585035 16187 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "240c1ed18e4b412f8997d62a51aa7b6a"
format_stamp: "Formatted at 2026-06-20 01:57:01 on dist-test-slave-wg14"
I20260620 01:57:01.593564 16187 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260620 01:57:01.599205 16205 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:01.601142 16187 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260620 01:57:01.601423 16187 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "240c1ed18e4b412f8997d62a51aa7b6a"
format_stamp: "Formatted at 2026-06-20 01:57:01 on dist-test-slave-wg14"
I20260620 01:57:01.601733 16187 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:01.637435 16187 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:01.645340 16187 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:01.645682 16187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:01.647315 16187 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:01.650159 16187 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:01.650313 16187 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:01.650465 16187 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:01.650523 16187 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:01.696084 16187 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:41081
I20260620 01:57:01.696127 16318 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:41081 every 8 connection(s)
I20260620 01:57:01.697974 16187 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:57:01.705066 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 16187
I20260620 01:57:01.705346 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 01:57:01 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920621, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:57:01 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920621, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:57:01 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920621, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:57:01.757974 16319 heartbeater.cc:344] Connected to a master server at 127.20.1.188:37811
I20260620 01:57:01.758054 16320 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35471
I20260620 01:57:01.759075 16320 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:01.759083 16319 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:01.759771 16321 heartbeater.cc:344] Connected to a master server at 127.20.1.189:37567
I20260620 01:57:01.760010 16321 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:01.761233 16320 heartbeater.cc:507] Master 127.20.1.190:35471 requested a full tablet report, sending...
I20260620 01:57:01.761371 16319 heartbeater.cc:507] Master 127.20.1.188:37811 requested a full tablet report, sending...
I20260620 01:57:01.761282 16321 heartbeater.cc:507] Master 127.20.1.189:37567 requested a full tablet report, sending...
I20260620 01:57:01.764945 15926 ts_manager.cc:194] Registered new tserver with Master: 240c1ed18e4b412f8997d62a51aa7b6a (127.20.1.129:41081)
I20260620 01:57:01.764941 16082 ts_manager.cc:194] Registered new tserver with Master: 240c1ed18e4b412f8997d62a51aa7b6a (127.20.1.129:41081)
I20260620 01:57:01.764933 16003 ts_manager.cc:194] Registered new tserver with Master: 240c1ed18e4b412f8997d62a51aa7b6a (127.20.1.129:41081)
I20260620 01:57:01.767550 15926 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:33133
WARNING: no policy specified for kudu/127.20.1.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:57:01.909081 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:57:02.421175 16333 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:02.421787 16333 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:02.421903 16333 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:02.433584 16333 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:02.433859 16333 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:57:02.449862 16333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:02.452119 16333 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:02.454149 16333 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:02.475208 16338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:02.475414 16339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:02.476411 16341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:02.477631 16333 server_base.cc:1061] running on GCE node
I20260620 01:57:02.478884 16333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:02.481570 16333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:02.485486 16333 hybrid_clock.cc:648] HybridClock initialized: now 1781920622484962 us; error 354 us; skew 500 ppm
I20260620 01:57:02.486310 16333 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:57:02 dist-test-slave-wg14 krb5kdc[15585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920622, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:57:02.491602 16333 init.cc:377] Logged in from keytab as kudu/127.20.1.130@KRBTEST.COM (short username kudu)
I20260620 01:57:02.494330 16333 webserver.cc:533] Webserver started at http://127.20.1.130:34993/ using document root <none> and password file <none>
I20260620 01:57:02.495455 16333 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:02.495627 16333 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:02.496102 16333 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:02.499354 16333 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/data/instance:
uuid: "e4c38bccb47c4d139947f898d3ffa7c2"
format_stamp: "Formatted at 2026-06-20 01:57:02 on dist-test-slave-wg14"
I20260620 01:57:02.500480 16333 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance:
uuid: "e4c38bccb47c4d139947f898d3ffa7c2"
format_stamp: "Formatted at 2026-06-20 01:57:02 on dist-test-slave-wg14"
I20260620 01:57:02.510021 16333 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.003s sys 0.005s
I20260620 01:57:02.515878 16348 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:02.518394 16333 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:57:02.518667 16333 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "e4c38bccb47c4d139947f898d3ffa7c2"
format_stamp: "Formatted at 2026-06-20 01:57:02 on dist-test-slave-wg14"
I20260620 01:57:02.518959 16333 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:02.540669 16333 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:02.547628 16333 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:02.548005 16333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:02.549278 16333 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:02.551687 16333 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:02.551846 16333 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:02.551994 16333 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:02.552054 16333 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:02.600950 16333 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:41009
I20260620 01:57:02.601055 16461 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:41009 every 8 connection(s)
I20260620 01:57:02.602887 16333 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:57:02.605803 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 16333
I20260620 01:57:02.606083 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
Jun 20 01:57:02 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920622, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:57:02 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920622, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:57:02.670428 16463 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35471
I20260620 01:57:02.671185 16463 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:02.673203 16463 heartbeater.cc:507] Master 127.20.1.190:35471 requested a full tablet report, sending...
Jun 20 01:57:02 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920622, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:57:02.674676 16462 heartbeater.cc:344] Connected to a master server at 127.20.1.188:37811
I20260620 01:57:02.674903 16462 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:02.675812 16462 heartbeater.cc:507] Master 127.20.1.188:37811 requested a full tablet report, sending...
I20260620 01:57:02.677610 15926 ts_manager.cc:194] Registered new tserver with Master: e4c38bccb47c4d139947f898d3ffa7c2 (127.20.1.130:41009)
I20260620 01:57:02.679344 16082 ts_manager.cc:194] Registered new tserver with Master: e4c38bccb47c4d139947f898d3ffa7c2 (127.20.1.130:41009)
I20260620 01:57:02.680009 15926 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:58405
I20260620 01:57:02.686383 16464 heartbeater.cc:344] Connected to a master server at 127.20.1.189:37567
I20260620 01:57:02.686640 16464 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:02.687265 16464 heartbeater.cc:507] Master 127.20.1.189:37567 requested a full tablet report, sending...
I20260620 01:57:02.689219 16003 ts_manager.cc:194] Registered new tserver with Master: e4c38bccb47c4d139947f898d3ffa7c2 (127.20.1.130:41009)
WARNING: no policy specified for kudu/127.20.1.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:57:02.741218 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.131: real 0.100s user 0.006s sys 0.020s
WARNING: no policy specified for HTTP/127.20.1.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.131@KRBTEST.COM" created.
I20260620 01:57:02.777747 16320 heartbeater.cc:499] Master 127.20.1.190:35471 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:57:02.813161 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:57:03.343039 16476 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:03.343689 16476 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:03.343852 16476 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:03.356539 16476 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:03.356801 16476 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:57:03.370724 16476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:03.372872 16476 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:03.374610 16476 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:03.388598 16481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:03.388813 16482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:03.390633 16485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:03.391352 16476 server_base.cc:1061] running on GCE node
I20260620 01:57:03.392132 16476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:03.394057 16476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:03.397668 16476 hybrid_clock.cc:648] HybridClock initialized: now 1781920623397432 us; error 140 us; skew 500 ppm
I20260620 01:57:03.398317 16476 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:57:03 dist-test-slave-wg14 krb5kdc[15585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920623, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:57:03.404071 16476 init.cc:377] Logged in from keytab as kudu/127.20.1.131@KRBTEST.COM (short username kudu)
I20260620 01:57:03.406637 16476 webserver.cc:533] Webserver started at http://127.20.1.131:39341/ using document root <none> and password file <none>
I20260620 01:57:03.408372 16476 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:03.408613 16476 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:03.409682 16476 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:03.413270 16476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/data/instance:
uuid: "a8fc2dc1fbaa420eaadc512eda60b0db"
format_stamp: "Formatted at 2026-06-20 01:57:03 on dist-test-slave-wg14"
I20260620 01:57:03.414404 16476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance:
uuid: "a8fc2dc1fbaa420eaadc512eda60b0db"
format_stamp: "Formatted at 2026-06-20 01:57:03 on dist-test-slave-wg14"
I20260620 01:57:03.424721 16476 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s
I20260620 01:57:03.431283 16492 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:03.433612 16476 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:57:03.433934 16476 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "a8fc2dc1fbaa420eaadc512eda60b0db"
format_stamp: "Formatted at 2026-06-20 01:57:03 on dist-test-slave-wg14"
I20260620 01:57:03.434264 16476 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:03.457050 16476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:03.464634 16476 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:03.465139 16476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:03.466905 16476 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:03.469844 16476 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:03.469988 16476 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:03.470127 16476 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:03.470196 16476 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:03.528702 16476 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:35659
I20260620 01:57:03.528891 16605 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:35659 every 8 connection(s)
I20260620 01:57:03.530730 16476 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:57:03.533641 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 16476
I20260620 01:57:03.533962 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
Jun 20 01:57:03 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920623, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:57:03 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920623, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:57:03 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920623, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:57:03.586490 16607 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35471
I20260620 01:57:03.586922 16608 heartbeater.cc:344] Connected to a master server at 127.20.1.189:37567
I20260620 01:57:03.586985 16606 heartbeater.cc:344] Connected to a master server at 127.20.1.188:37811
I20260620 01:57:03.587332 16608 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:03.587322 16607 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:03.587304 16606 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:03.589191 16606 heartbeater.cc:507] Master 127.20.1.188:37811 requested a full tablet report, sending...
I20260620 01:57:03.589308 16608 heartbeater.cc:507] Master 127.20.1.189:37567 requested a full tablet report, sending...
I20260620 01:57:03.591104 16607 heartbeater.cc:507] Master 127.20.1.190:35471 requested a full tablet report, sending...
I20260620 01:57:03.592939 16003 ts_manager.cc:194] Registered new tserver with Master: a8fc2dc1fbaa420eaadc512eda60b0db (127.20.1.131:35659)
I20260620 01:57:03.593855 15926 ts_manager.cc:194] Registered new tserver with Master: a8fc2dc1fbaa420eaadc512eda60b0db (127.20.1.131:35659)
I20260620 01:57:03.594015 16081 ts_manager.cc:194] Registered new tserver with Master: a8fc2dc1fbaa420eaadc512eda60b0db (127.20.1.131:35659)
I20260620 01:57:03.595369 15926 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:53735
I20260620 01:57:03.605732 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:57:03.620627 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:57:03.620858 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 15894
I20260620 01:57:03.623524 15957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:03.685887 16463 heartbeater.cc:499] Master 127.20.1.190:35471 was elected leader, sending a full tablet report...
I20260620 01:57:03.758517 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 15894
W20260620 01:57:03.781477 15989 connection.cc:573] server connection from 127.0.0.1:41340 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:57:03.782630 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:35471
--webserver_interface=127.20.1.190
--webserver_port=34379
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:57:03.783726 16320 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:35471 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:35471: connect: Connection refused (error 111)
W20260620 01:57:04.313714 16618 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:04.314116 16618 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:04.314265 16618 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:04.325335 16618 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:04.325541 16618 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:04.325634 16618 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:04.325711 16618 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:04.340384 16618 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:35471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=34379
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:04.344275 16618 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:04.346221 16618 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:04.360754 16623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:04.362018 16624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:04.363554 16626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:04.364509 16618 server_base.cc:1061] running on GCE node
I20260620 01:57:04.365593 16618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:04.368393 16618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:04.372056 16618 hybrid_clock.cc:648] HybridClock initialized: now 1781920624371700 us; error 218 us; skew 500 ppm
I20260620 01:57:04.372825 16618 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:57:04 dist-test-slave-wg14 krb5kdc[15585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920624, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:57:04.379055 16618 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:57:04.383003 16618 webserver.cc:533] Webserver started at http://127.20.1.190:34379/ using document root <none> and password file <none>
I20260620 01:57:04.384702 16618 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:04.384912 16618 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:04.395107 16618 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 01:57:04.403563 16633 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:04.406289 16618 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260620 01:57:04.406603 16618 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "5e1a5c80089949c18d5b0336c188a008"
format_stamp: "Formatted at 2026-06-20 01:56:58 on dist-test-slave-wg14"
I20260620 01:57:04.407924 16618 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:04.451352 16618 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:04.459381 16618 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:04.459929 16618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:04.491073 16618 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:35471
I20260620 01:57:04.491086 16686 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:35471 every 8 connection(s)
I20260620 01:57:04.493732 16618 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:57:04.495254 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 16618
I20260620 01:57:04.495815 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 16187
I20260620 01:57:04.505529 16687 sys_catalog.cc:263] Verifying existing consensus state
Jun 20 01:57:04 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 01:57:04.509869 16687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Bootstrap starting.
I20260620 01:57:04.519110 16314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:04.536687 16687 log.cc:826] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:04.550361 16687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:57:04.551056 16687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Bootstrap complete.
I20260620 01:57:04.562775 16687 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:04.564244 16687 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1a5c80089949c18d5b0336c188a008, State: Initialized, Role: FOLLOWER
I20260620 01:57:04.565644 16687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:04.568256 16693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:04.568825 16693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:04.569161 16687 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:04.590425 16705 catalog_manager.cc:1294] Loaded cluster ID: c67ebe9c0bd849748e5f573512cf92ac
I20260620 01:57:04.592572 16705 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: loading cluster ID for follower catalog manager: success
I20260620 01:57:04.599088 16607 heartbeater.cc:499] Master 127.20.1.190:35471 was elected leader, sending a full tablet report...
I20260620 01:57:04.602173 16705 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: acquiring CA information for follower catalog manager: success
I20260620 01:57:04.607053 16705 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:57:04.631389 16649 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "a8fc2dc1fbaa420eaadc512eda60b0db" instance_seqno: 1781920623515459) as {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:49629; Asking this server to re-register.
I20260620 01:57:04.633112 16607 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:04.635756 16649 ts_manager.cc:194] Registered new tserver with Master: a8fc2dc1fbaa420eaadc512eda60b0db (127.20.1.131:35659)
I20260620 01:57:04.661307 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 16187
I20260620 01:57:04.669883 16618 master_runner.cc:264] No leader master found from master 5e1a5c80089949c18d5b0336c188a008
I20260620 01:57:04.670101 16618 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:57:04.680920 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:41081
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--webserver_port=35687
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:57:04.772706 16649 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e4c38bccb47c4d139947f898d3ffa7c2" instance_seqno: 1781920622589105) as {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:60165; Asking this server to re-register.
I20260620 01:57:04.774382 16463 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:04.776510 16649 ts_manager.cc:194] Registered new tserver with Master: e4c38bccb47c4d139947f898d3ffa7c2 (127.20.1.130:41009)
I20260620 01:57:04.781338 16715 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1a5c80089949c18d5b0336c188a008)
I20260620 01:57:04.781687 16715 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:04.784379 16715 leader_election.cc:290] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1a5c80089949c18d5b0336c188a008 (127.20.1.190:35471), 7a004ebbc3c54fbe98351e428a62af87 (127.20.1.189:37567)
I20260620 01:57:04.785449 16011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b772652276745cabab57b0bd8ef5983" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a004ebbc3c54fbe98351e428a62af87" is_pre_election: true
I20260620 01:57:04.798595 16660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b772652276745cabab57b0bd8ef5983" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e1a5c80089949c18d5b0336c188a008" is_pre_election: true
I20260620 01:57:04.800675 16660 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b772652276745cabab57b0bd8ef5983 in term 1.
I20260620 01:57:04.801916 16066 leader_election.cc:304] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e1a5c80089949c18d5b0336c188a008, 9b772652276745cabab57b0bd8ef5983; no voters: 7a004ebbc3c54fbe98351e428a62af87
I20260620 01:57:04.802603 16715 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:57:04.802875 16715 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e1a5c80089949c18d5b0336c188a008)
I20260620 01:57:04.803093 16715 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:57:04.806663 16715 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:04.807814 16715 leader_election.cc:290] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [CANDIDATE]: Term 2 election: Requested vote from peers 5e1a5c80089949c18d5b0336c188a008 (127.20.1.190:35471), 7a004ebbc3c54fbe98351e428a62af87 (127.20.1.189:37567)
I20260620 01:57:04.809106 16011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b772652276745cabab57b0bd8ef5983" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a004ebbc3c54fbe98351e428a62af87"
I20260620 01:57:04.809300 16660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b772652276745cabab57b0bd8ef5983" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e1a5c80089949c18d5b0336c188a008"
I20260620 01:57:04.810367 16660 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:57:04.820439 16660 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b772652276745cabab57b0bd8ef5983 in term 2.
I20260620 01:57:04.821302 16066 leader_election.cc:304] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e1a5c80089949c18d5b0336c188a008, 9b772652276745cabab57b0bd8ef5983; no voters: 7a004ebbc3c54fbe98351e428a62af87
I20260620 01:57:04.821938 16715 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 01:57:04.822528 16715 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 2 LEADER]: Becoming Leader. State: Replica: 9b772652276745cabab57b0bd8ef5983, State: Running, Role: LEADER
I20260620 01:57:04.823237 16715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:04.826931 16717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b772652276745cabab57b0bd8ef5983. Latest consensus state: current_term: 2 leader_uuid: "9b772652276745cabab57b0bd8ef5983" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:04.827540 16717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:04.829527 16719 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:57:04.833875 16719 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:57:04.835855 16719 catalog_manager.cc:1294] Loaded cluster ID: c67ebe9c0bd849748e5f573512cf92ac
I20260620 01:57:04.836342 16719 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:57:04.838861 16719 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:57:04.840879 16719 catalog_manager.cc:6080] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983: Loaded TSK: 0
I20260620 01:57:04.842491 16719 catalog_manager.cc:1549] Initializing in-progress tserver states...
W20260620 01:57:05.223893 16713 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:05.224340 16713 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:05.224481 16713 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:05.234532 16713 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:05.234781 16713 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:05.248812 16713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:41081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=35687
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:05.250933 16713 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:05.252717 16713 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260620 01:57:05.260638 16723 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1a5c80089949c18d5b0336c188a008)
I20260620 01:57:05.260943 16723 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:05.261734 16660 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 2 FOLLOWER]: Refusing update from remote peer 9b772652276745cabab57b0bd8ef5983: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:57:05.262499 16723 leader_election.cc:290] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1a5c80089949c18d5b0336c188a008 (127.20.1.190:35471), 9b772652276745cabab57b0bd8ef5983 (127.20.1.188:37811)
I20260620 01:57:05.263077 16717 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:57:05.263849 16092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a004ebbc3c54fbe98351e428a62af87" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9b772652276745cabab57b0bd8ef5983" is_pre_election: true
I20260620 01:57:05.270547 16726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b772652276745cabab57b0bd8ef5983. Latest consensus state: current_term: 2 leader_uuid: "9b772652276745cabab57b0bd8ef5983" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:05.271060 16726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:05.274017 16659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a004ebbc3c54fbe98351e428a62af87" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e1a5c80089949c18d5b0336c188a008" is_pre_election: true
W20260620 01:57:05.274511 16728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:05.275209 15987 leader_election.cc:304] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a004ebbc3c54fbe98351e428a62af87; no voters: 5e1a5c80089949c18d5b0336c188a008, 9b772652276745cabab57b0bd8ef5983
I20260620 01:57:05.276301 16723 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 01:57:05.277802 16727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:05.280884 16715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9b772652276745cabab57b0bd8ef5983" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:05.280886 16726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9b772652276745cabab57b0bd8ef5983" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:05.281282 16715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:05.281414 16726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:05.281584 16011 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 1 FOLLOWER]: Advancing to term 2
W20260620 01:57:05.282694 16733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:05.282631 16713 server_base.cc:1061] running on GCE node
I20260620 01:57:05.283888 16713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:05.285672 16713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:05.285989 16011 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 2 FOLLOWER]: Refusing update from remote peer 9b772652276745cabab57b0bd8ef5983: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 01:57:05.287225 16735 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:57:05.289304 16713 hybrid_clock.cc:648] HybridClock initialized: now 1781920625288965 us; error 239 us; skew 500 ppm
I20260620 01:57:05.289974 16713 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:05.291378 16723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b772652276745cabab57b0bd8ef5983. Latest consensus state: current_term: 2 leader_uuid: "9b772652276745cabab57b0bd8ef5983" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:05.291759 16723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:05.291321 16739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9b772652276745cabab57b0bd8ef5983" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:05.292330 16739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 01:57:05 dist-test-slave-wg14 krb5kdc[15585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920625, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:57:05.295632 16713 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:57:05.298784 16713 webserver.cc:533] Webserver started at http://127.20.1.129:35687/ using document root <none> and password file <none>
I20260620 01:57:05.300391 16713 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:05.300629 16713 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:05.311419 16713 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 01:57:05.316270 16744 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:05.318995 16713 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.002s
I20260620 01:57:05.319386 16713 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "240c1ed18e4b412f8997d62a51aa7b6a"
format_stamp: "Formatted at 2026-06-20 01:57:01 on dist-test-slave-wg14"
I20260620 01:57:05.320482 16713 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:05.383318 16713 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:05.388258 16713 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:05.388608 16713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:05.389863 16713 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:05.392146 16713 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:05.392275 16713 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:05.392400 16713 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:05.392482 16713 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:05.439030 16713 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:41081
I20260620 01:57:05.439085 16858 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:41081 every 8 connection(s)
I20260620 01:57:05.441188 16713 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:57:05.442862 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 16713
Jun 20 01:57:05 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Jun 20 01:57:05 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920625, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:57:05 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920625, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:57:05 dist-test-slave-wg14 krb5kdc[15585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920625, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:57:05.502746 16861 heartbeater.cc:344] Connected to a master server at 127.20.1.189:37567
I20260620 01:57:05.503356 16861 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:05.504900 16861 heartbeater.cc:507] Master 127.20.1.189:37567 requested a full tablet report, sending...
I20260620 01:57:05.507550 16860 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35471
I20260620 01:57:05.507810 16860 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:05.507817 16003 ts_manager.cc:194] Re-registered known tserver with Master: 240c1ed18e4b412f8997d62a51aa7b6a (127.20.1.129:41081)
I20260620 01:57:05.508632 16860 heartbeater.cc:507] Master 127.20.1.190:35471 requested a full tablet report, sending...
I20260620 01:57:05.510483 16649 ts_manager.cc:194] Registered new tserver with Master: 240c1ed18e4b412f8997d62a51aa7b6a (127.20.1.129:41081)
I20260620 01:57:05.518759 16859 heartbeater.cc:344] Connected to a master server at 127.20.1.188:37811
I20260620 01:57:05.518999 16859 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:05.519629 16859 heartbeater.cc:507] Master 127.20.1.188:37811 requested a full tablet report, sending...
I20260620 01:57:05.521904 16081 ts_manager.cc:194] Re-registered known tserver with Master: 240c1ed18e4b412f8997d62a51aa7b6a (127.20.1.129:41081)
I20260620 01:57:05.525206 16081 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:53979
I20260620 01:57:05.671612 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 16713
W20260620 01:57:05.674121 16873 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:57:05.667430 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.20.1.129:41081)
0620 01:57:05.668675 (+ 1245us) server_negotiation.cc:207] Beginning negotiation
0620 01:57:05.668684 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0620 01:57:05.668726 (+ 42us) server_negotiation.cc:408] Connection header received
0620 01:57:05.668816 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:57:05.668824 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:57:05.669100 (+ 276us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:57:05.669420 (+ 320us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:57:05.671115 (+ 1695us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43964: BlockingRecv error: recv got EOF from 127.0.0.1:43964 (error 108)
Metrics: {"server-negotiator.queue_time_us":1076,"thread_start_us":857,"threads_started":1}
W20260620 01:57:05.675720 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 16713: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:41081: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:57:05.675918 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 16713
I20260620 01:57:05.693732 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:41081
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--webserver_port=35687
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:57:05.793365 16735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9b772652276745cabab57b0bd8ef5983" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:05.794044 16735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: This master's current role is: LEADER
W20260620 01:57:06.168190 16875 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:06.168664 16875 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:06.168814 16875 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:57:06.168905 16875 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:06.182349 16875 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:06.182669 16875 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:06.197857 16875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:41081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=35687
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:06.201177 16875 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:06.203207 16875 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:06.219164 16884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:06.218475 16883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:06.222175 16886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:06.222175 16875 server_base.cc:1061] running on GCE node
I20260620 01:57:06.223274 16875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:06.224969 16875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:06.228681 16875 hybrid_clock.cc:648] HybridClock initialized: now 1781920626228410 us; error 149 us; skew 500 ppm
I20260620 01:57:06.229193 16875 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:57:06 dist-test-slave-wg14 krb5kdc[15585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920626, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:57:06.233973 16875 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:57:06.238124 16875 webserver.cc:533] Webserver started at http://127.20.1.129:35687/ using document root <none> and password file <none>
I20260620 01:57:06.239288 16875 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:06.239440 16875 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:06.259913 16875 fs_manager.cc:714] Time spent opening directory manager: real 0.017s user 0.010s sys 0.007s
I20260620 01:57:06.264249 16893 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:06.266981 16875 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:57:06.267517 16875 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "240c1ed18e4b412f8997d62a51aa7b6a"
format_stamp: "Formatted at 2026-06-20 01:57:01 on dist-test-slave-wg14"
I20260620 01:57:06.268603 16875 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:06.292379 16875 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:06.299185 16875 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:06.299611 16875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:06.301323 16875 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:06.303829 16875 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:06.304003 16875 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:06.304181 16875 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:06.304258 16875 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:57:06.304795 16875 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:57:06.304886 16875 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:57:06.305011 16875 fault_injection.cc:59] injecting fault for kudu (pid 16875): FLAGS_fault_before_start (process will exit)
I20260620 01:57:06.327975 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:41081
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--webserver_port=35687
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:57:06.601217 16606 heartbeater.cc:499] Master 127.20.1.188:37811 was elected leader, sending a full tablet report...
I20260620 01:57:06.689476 16462 heartbeater.cc:499] Master 127.20.1.188:37811 was elected leader, sending a full tablet report...
W20260620 01:57:06.789503 16901 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:06.790019 16901 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:06.790185 16901 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:06.801375 16901 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:06.801919 16901 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:06.815361 16901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:43823,127.20.1.149:46357,127.20.1.150:36305,127.20.1.151:39421,127.20.1.152:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:41081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=35687
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:06.817622 16901 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:06.819633 16901 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:06.834542 16907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:06.835201 16906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:06.840831 16909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:06.841472 16901 server_base.cc:1061] running on GCE node
I20260620 01:57:06.842276 16901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:06.844262 16901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:06.846201 16901 hybrid_clock.cc:648] HybridClock initialized: now 1781920626846063 us; error 79 us; skew 500 ppm
I20260620 01:57:06.846696 16901 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:57:06 dist-test-slave-wg14 krb5kdc[15585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920626, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:57:06.851646 16901 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:57:06.855352 16901 webserver.cc:533] Webserver started at http://127.20.1.129:35687/ using document root <none> and password file <none>
I20260620 01:57:06.856388 16901 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:06.856542 16901 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:06.865335 16901 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.008s sys 0.001s
I20260620 01:57:06.870409 16916 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:06.873363 16901 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.004s
I20260620 01:57:06.873657 16901 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "240c1ed18e4b412f8997d62a51aa7b6a"
format_stamp: "Formatted at 2026-06-20 01:57:01 on dist-test-slave-wg14"
I20260620 01:57:06.875047 16901 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:06.894685 16901 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:06.902920 16901 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:06.903396 16901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:06.904826 16901 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:06.909406 16901 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:06.909605 16901 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:06.909752 16901 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:06.909845 16901 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:06.965698 16901 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:41081
I20260620 01:57:06.965746 17029 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:41081 every 8 connection(s)
I20260620 01:57:06.967792 16901 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:57:06.970723 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 16901
I20260620 01:57:06.971195 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 16901
W20260620 01:57:06.999066 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 16901: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:41081: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:57:06.999338 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 16901
W20260620 01:57:07.023532 17035 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:57:06.972972 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:37715 (local address 127.20.1.189:37567)
0620 01:57:06.992253 (+ 19281us) server_negotiation.cc:207] Beginning negotiation
0620 01:57:06.992263 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0620 01:57:06.993238 (+ 975us) server_negotiation.cc:408] Connection header received
0620 01:57:06.993496 (+ 258us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:57:06.993504 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:57:06.993623 (+ 119us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:57:06.993793 (+ 170us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:57:06.999229 (+ 5436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:57:07.001796 (+ 2567us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:57:07.022404 (+ 20608us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:37715: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":18898,"thread_start_us":18663,"threads_started":1}
W20260620 01:57:07.023977 17036 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:57:06.973357 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:59919 (local address 127.20.1.188:37811)
0620 01:57:06.987992 (+ 14635us) server_negotiation.cc:207] Beginning negotiation
0620 01:57:06.988004 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0620 01:57:06.988058 (+ 54us) server_negotiation.cc:408] Connection header received
0620 01:57:06.990802 (+ 2744us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:57:06.990812 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:57:06.990998 (+ 186us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:57:06.991236 (+ 238us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:57:06.995681 (+ 4445us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:57:06.997336 (+ 1655us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:57:07.022470 (+ 25134us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:59919: BlockingRecv error: recv got EOF from 127.20.1.129:59919 (error 108)
Metrics: {"server-negotiator.queue_time_us":14185,"thread_start_us":13972,"threads_started":1}
W20260620 01:57:07.024025 17037 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:57:06.973796 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:33819 (local address 127.20.1.190:35471)
0620 01:57:06.988779 (+ 14983us) server_negotiation.cc:207] Beginning negotiation
0620 01:57:06.988788 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0620 01:57:06.994091 (+ 5303us) server_negotiation.cc:408] Connection header received
0620 01:57:06.994321 (+ 230us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:57:06.994331 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:57:06.994467 (+ 136us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:57:06.994662 (+ 195us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:57:06.997858 (+ 3196us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:57:07.001090 (+ 3232us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:57:07.022404 (+ 21314us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:33819: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":14723,"thread_start_us":14569,"threads_started":1}
I20260620 01:57:07.024333 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 16333
W20260620 01:57:07.028503 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 16333: Not authorized: Client connection negotiation failed: client connection to 127.20.1.130:41009: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:57:07.028741 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 16333
I20260620 01:57:07.046887 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 16476
W20260620 01:57:07.051851 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 16476: Not authorized: Client connection negotiation failed: client connection to 127.20.1.131:35659: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:57:07.052071 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 16476
I20260620 01:57:07.070326 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 16618
W20260620 01:57:07.073233 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 16618: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:35471: server requires authentication, but client does not have Kerberos credentials available
W20260620 01:57:07.073321 17037 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:57:07.071687 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:33490 (local address 127.20.1.190:35471)
0620 01:57:07.071894 (+ 207us) server_negotiation.cc:207] Beginning negotiation
0620 01:57:07.071900 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0620 01:57:07.071918 (+ 18us) server_negotiation.cc:408] Connection header received
0620 01:57:07.071986 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:57:07.071991 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:57:07.072068 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:57:07.072169 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:57:07.073106 (+ 937us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33490: BlockingRecv error: recv got EOF from 127.0.0.1:33490 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 01:57:07.073459 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 16618
I20260620 01:57:07.094458 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 15971
I20260620 01:57:07.096518 16033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:07.232944 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 15971
I20260620 01:57:07.253680 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 16048
I20260620 01:57:07.255271 16113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:07.394122 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 16048
Shutting down hive metastore.
2026-06-20T01:57:07Z chronyd exiting
2026-06-20T01:57:07Z chronyd exiting
2026-06-20T01:57:07Z chronyd exiting
2026-06-20T01:57:07Z chronyd exiting
2026-06-20T01:57:07Z chronyd exiting
Jun 20 01:57:07 dist-test-slave-wg14 krb5kdc[17054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 01:57:07 dist-test-slave-wg14 krb5kdc[17054](info): set up 2 sockets
Jun 20 01:57:07 dist-test-slave-wg14 krb5kdc[17054](info): commencing operation
krb5kdc: starting...
W20260620 01:57:09.887543 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.034s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 01:57:10 dist-test-slave-wg14 krb5kdc[17054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T01:57:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:57:10Z Disabled control of system clock
2026-06-20T01:57:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:57:10Z Disabled control of system clock
2026-06-20T01:57:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:57:10Z Disabled control of system clock
2026-06-20T01:57:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:57:10Z Disabled control of system clock
2026-06-20T01:57:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:57:10Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "hive/127.0.0.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf
2026-06-20 01:57:11: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Jun 20 01:57:14 dist-test-slave-wg14 krb5kdc[17054](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920634, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 01:57:20 dist-test-slave-wg14 krb5kdc[17054](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920640, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 01:57:20 dist-test-slave-wg14 krb5kdc[17054](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920640, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.190@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.190@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:57:22.419427 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:35471
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:57:22.862632 17258 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:22.863049 17258 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:22.863199 17258 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:22.873857 17258 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:22.874010 17258 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:22.874070 17258 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:22.874121 17258 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:22.888343 17258 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:35471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:22.890481 17258 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:22.892351 17258 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:22.903764 17264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:22.903782 17263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:22.903764 17266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:22.904898 17258 server_base.cc:1061] running on GCE node
I20260620 01:57:22.905968 17258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:22.908541 17258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:22.912158 17258 hybrid_clock.cc:648] HybridClock initialized: now 1781920642911866 us; error 187 us; skew 500 ppm
I20260620 01:57:22.912899 17258 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:57:22 dist-test-slave-wg14 krb5kdc[17054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920642, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:57:22.918556 17258 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:57:22.921555 17258 webserver.cc:533] Webserver started at http://127.20.1.190:42791/ using document root <none> and password file <none>
I20260620 01:57:22.922622 17258 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:22.922811 17258 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:22.931293 17258 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.003s
I20260620 01:57:22.937974 17273 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:22.941828 17258 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.005s sys 0.001s
I20260620 01:57:22.942232 17258 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "5e1a5c80089949c18d5b0336c188a008"
format_stamp: "Formatted at 2026-06-20 01:56:58 on dist-test-slave-wg14"
I20260620 01:57:22.943730 17258 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:22.978397 17258 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:22.986804 17258 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:22.987212 17258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:23.038308 17258 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:35471
I20260620 01:57:23.038309 17325 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:35471 every 8 connection(s)
I20260620 01:57:23.040814 17258 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:57:23.047652 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 17258
I20260620 01:57:23.047933 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:57:23.052723 17326 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:57:23.062132 17326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Bootstrap starting.
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.189@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:57:23.091353 17326 log.cc:826] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:23.109468 17326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:57:23.110231 17326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Bootstrap complete.
I20260620 01:57:23.120060 17326 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:23.121126 17326 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1a5c80089949c18d5b0336c188a008, State: Initialized, Role: FOLLOWER
I20260620 01:57:23.122576 17326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:23.124032 17331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:23.124593 17331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:23.125545 17326 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: configured and running, proceeding with master startup.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:57:23.138928 17342 catalog_manager.cc:1294] Loaded cluster ID: c67ebe9c0bd849748e5f573512cf92ac
I20260620 01:57:23.139285 17342 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: loading cluster ID for follower catalog manager: success
I20260620 01:57:23.146162 17342 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: acquiring CA information for follower catalog manager: success
I20260620 01:57:23.153915 17342 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:57:23.169523 17258 master_runner.cc:187] Error getting master registration for 127.20.1.189:37567: Network error: Client connection negotiation failed: client connection to 127.20.1.189:37567: connect: Connection refused (error 111),
I20260620 01:57:23.175199 17258 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.189@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:57:23.223918 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:37567
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:57:23.735072 17348 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:23.735569 17348 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:23.735672 17348 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:23.745771 17348 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:23.745967 17348 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:23.746039 17348 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:23.746100 17348 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:23.760596 17348 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:37567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:23.762926 17348 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:23.765209 17348 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:23.776717 17356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:23.776940 17354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:23.776666 17353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:23.779444 17348 server_base.cc:1061] running on GCE node
I20260620 01:57:23.780740 17348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:23.783243 17348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:23.784850 17348 hybrid_clock.cc:648] HybridClock initialized: now 1781920643784684 us; error 93 us; skew 500 ppm
I20260620 01:57:23.785535 17348 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:57:23 dist-test-slave-wg14 krb5kdc[17054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920643, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:57:23.790720 17348 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 01:57:23.793637 17348 webserver.cc:533] Webserver started at http://127.20.1.189:34231/ using document root <none> and password file <none>
I20260620 01:57:23.794742 17348 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:23.794945 17348 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:23.804838 17348 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.005s sys 0.003s
I20260620 01:57:23.809926 17363 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:23.812390 17348 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:57:23.812685 17348 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "7a004ebbc3c54fbe98351e428a62af87"
format_stamp: "Formatted at 2026-06-20 01:56:59 on dist-test-slave-wg14"
I20260620 01:57:23.813735 17348 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:23.848448 17348 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:23.854730 17348 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:23.855082 17348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:23.883716 17414 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:37567 every 8 connection(s)
I20260620 01:57:23.883870 17348 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:37567
I20260620 01:57:23.886729 17348 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:57:23.889644 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 17348
I20260620 01:57:23.889870 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:57:23.896440 17415 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:57:23.902936 17415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87: Bootstrap starting.
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.188@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:57:23.931582 17415 log.cc:826] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:23.949107 17415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:57:23.949929 17415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87: Bootstrap complete.
I20260620 01:57:23.963768 17415 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:23.965397 17415 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a004ebbc3c54fbe98351e428a62af87, State: Initialized, Role: FOLLOWER
I20260620 01:57:23.966547 17415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:23.967844 17420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:23.968303 17420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:23.969233 17415 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: configured and running, proceeding with master startup.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:57:23.985229 17431 catalog_manager.cc:1294] Loaded cluster ID: c67ebe9c0bd849748e5f573512cf92ac
I20260620 01:57:23.985529 17431 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87: loading cluster ID for follower catalog manager: success
I20260620 01:57:23.991075 17431 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87: acquiring CA information for follower catalog manager: success
I20260620 01:57:23.996946 17431 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.188@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:57:24.042899 17348 master_runner.cc:187] Error getting master registration for 127.20.1.188:37811: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37811: connect: Connection refused (error 111),
I20260620 01:57:24.052299 17348 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:57:24.095261 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.188: real 0.117s user 0.005s sys 0.029s
I20260620 01:57:24.096669 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:37811
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:57:24.620036 17438 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:24.620476 17438 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:24.620616 17438 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:24.632642 17438 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:24.632817 17438 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:24.632885 17438 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:24.632946 17438 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:24.648342 17438 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:37811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:24.650411 17438 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:24.652642 17438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:24.664790 17445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:24.665400 17444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:24.667084 17447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:24.667919 17438 server_base.cc:1061] running on GCE node
I20260620 01:57:24.669886 17438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:24.672655 17438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:24.676244 17438 hybrid_clock.cc:648] HybridClock initialized: now 1781920644676002 us; error 159 us; skew 500 ppm
I20260620 01:57:24.676858 17438 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:57:24 dist-test-slave-wg14 krb5kdc[17054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920644, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:57:24.681793 17438 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 01:57:24.684826 17438 webserver.cc:533] Webserver started at http://127.20.1.188:39655/ using document root <none> and password file <none>
I20260620 01:57:24.686165 17438 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:24.686445 17438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:24.696174 17438 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.003s sys 0.004s
I20260620 01:57:24.701634 17454 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:24.704260 17438 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260620 01:57:24.704541 17438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "9b772652276745cabab57b0bd8ef5983"
format_stamp: "Formatted at 2026-06-20 01:56:59 on dist-test-slave-wg14"
I20260620 01:57:24.705515 17438 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:24.738924 17438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:24.745025 17438 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:24.745394 17438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:24.775327 17438 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:37811
I20260620 01:57:24.775202 17505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:37811 every 8 connection(s)
I20260620 01:57:24.777972 17438 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:57:24.780676 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 17438
I20260620 01:57:24.780965 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:57:24.791504 17506 sys_catalog.cc:263] Verifying existing consensus state
Jun 20 01:57:24 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:57:24.800671 17512 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:57:24.801131 17512 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:24.802459 17506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983: Bootstrap starting.
I20260620 01:57:24.815343 17512 leader_election.cc:290] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a004ebbc3c54fbe98351e428a62af87 (127.20.1.189:37567), 9b772652276745cabab57b0bd8ef5983 (127.20.1.188:37811)
Jun 20 01:57:24 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:57:24.833257 17506 log.cc:826] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:24.836638 17390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e1a5c80089949c18d5b0336c188a008" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a004ebbc3c54fbe98351e428a62af87" is_pre_election: true
I20260620 01:57:24.837426 17390 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1a5c80089949c18d5b0336c188a008 in term 2.
Jun 20 01:57:24 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920642, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:57:24.838642 17275 leader_election.cc:304] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e1a5c80089949c18d5b0336c188a008, 7a004ebbc3c54fbe98351e428a62af87; no voters:
I20260620 01:57:24.839268 17512 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 01:57:24.839435 17512 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:57:24.839524 17512 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:57:24.846434 17512 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:24.848240 17390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e1a5c80089949c18d5b0336c188a008" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a004ebbc3c54fbe98351e428a62af87"
I20260620 01:57:24.848495 17390 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:57:24.853106 17506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
Jun 20 01:57:24 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:57:24.854103 17506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983: Bootstrap complete.
I20260620 01:57:24.855175 17390 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1a5c80089949c18d5b0336c188a008 in term 3.
I20260620 01:57:24.855422 17512 leader_election.cc:290] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [CANDIDATE]: Term 3 election: Requested vote from peers 7a004ebbc3c54fbe98351e428a62af87 (127.20.1.189:37567), 9b772652276745cabab57b0bd8ef5983 (127.20.1.188:37811)
I20260620 01:57:24.855865 17275 leader_election.cc:304] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e1a5c80089949c18d5b0336c188a008, 7a004ebbc3c54fbe98351e428a62af87; no voters:
I20260620 01:57:24.856297 17512 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 01:57:24.857836 17512 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 3 LEADER]: Becoming Leader. State: Replica: 5e1a5c80089949c18d5b0336c188a008, State: Running, Role: LEADER
I20260620 01:57:24.858664 17512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:24.863360 17518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e1a5c80089949c18d5b0336c188a008. Latest consensus state: current_term: 3 leader_uuid: "5e1a5c80089949c18d5b0336c188a008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:24.864038 17518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:24.869249 17521 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:57:24.868575 17506 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:24.870015 17506 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b772652276745cabab57b0bd8ef5983, State: Initialized, Role: FOLLOWER
I20260620 01:57:24.871269 17506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:24.872845 17521 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:57:24.873165 17522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:24.873937 17522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:24.874233 17521 catalog_manager.cc:1294] Loaded cluster ID: c67ebe9c0bd849748e5f573512cf92ac
I20260620 01:57:24.874357 17521 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:57:24.874573 17506 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:24.874617 17481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e1a5c80089949c18d5b0336c188a008" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9b772652276745cabab57b0bd8ef5983" is_pre_election: true
I20260620 01:57:24.875434 17481 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1a5c80089949c18d5b0336c188a008 in term 2.
I20260620 01:57:24.874644 17480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e1a5c80089949c18d5b0336c188a008" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9b772652276745cabab57b0bd8ef5983"
I20260620 01:57:24.876210 17521 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:57:24.876202 17480 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:57:24.877624 17521 catalog_manager.cc:6080] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Loaded TSK: 0
I20260620 01:57:24.879321 17521 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:57:24.883174 17480 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1a5c80089949c18d5b0336c188a008 in term 3.
I20260620 01:57:24.903311 17533 catalog_manager.cc:1294] Loaded cluster ID: c67ebe9c0bd849748e5f573512cf92ac
I20260620 01:57:24.903472 17533 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983: loading cluster ID for follower catalog manager: success
I20260620 01:57:24.909651 17533 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983: acquiring CA information for follower catalog manager: success
Jun 20 01:57:24 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920644, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:57:24.920282 17533 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.129@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.129@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:57:25.044392 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:57:25.285300 17390 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 3 FOLLOWER]: Refusing update from remote peer 5e1a5c80089949c18d5b0336c188a008: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:57:25.287412 17518 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:57:25.307672 17541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e1a5c80089949c18d5b0336c188a008. Latest consensus state: current_term: 3 leader_uuid: "5e1a5c80089949c18d5b0336c188a008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:25.308339 17541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:25.322399 17518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "5e1a5c80089949c18d5b0336c188a008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:25.322773 17518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:25.324921 17541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5e1a5c80089949c18d5b0336c188a008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:25.325415 17541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:25.325718 17480 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 3 FOLLOWER]: Refusing update from remote peer 5e1a5c80089949c18d5b0336c188a008: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 01:57:25.327603 17518 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:57:25.343453 17522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e1a5c80089949c18d5b0336c188a008. Latest consensus state: current_term: 3 leader_uuid: "5e1a5c80089949c18d5b0336c188a008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:25.344035 17522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:25.342005 17548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5e1a5c80089949c18d5b0336c188a008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:25.344383 17548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:57:25.605885 17540 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:25.606559 17540 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:25.606763 17540 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:25.618572 17540 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:25.618803 17540 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:25.634114 17540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:25.636193 17540 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:25.638072 17540 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:25.652828 17553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:25.653909 17554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:25.654417 17556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:25.654703 17540 server_base.cc:1061] running on GCE node
I20260620 01:57:25.655668 17540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:25.657274 17540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:25.660774 17540 hybrid_clock.cc:648] HybridClock initialized: now 1781920645660501 us; error 170 us; skew 500 ppm
I20260620 01:57:25.661392 17540 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:57:25 dist-test-slave-wg14 krb5kdc[17054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920645, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:57:25.667263 17540 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:57:25.669879 17540 webserver.cc:533] Webserver started at http://127.20.1.129:42223/ using document root <none> and password file <none>
I20260620 01:57:25.671094 17540 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:25.671617 17540 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:25.681005 17540 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 01:57:25.686630 17563 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:25.689733 17540 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260620 01:57:25.690229 17540 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "240c1ed18e4b412f8997d62a51aa7b6a"
format_stamp: "Formatted at 2026-06-20 01:57:01 on dist-test-slave-wg14"
I20260620 01:57:25.691710 17540 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:25.721539 17540 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:25.728713 17540 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:25.729090 17540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:25.730544 17540 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:25.733057 17540 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:25.733223 17540 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:25.733350 17540 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:25.733434 17540 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:25.782935 17540 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:41177
I20260620 01:57:25.783031 17676 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:41177 every 8 connection(s)
I20260620 01:57:25.785164 17540 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:57:25.786020 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 17540
I20260620 01:57:25.786295 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 01:57:25 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920645, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:57:25 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920645, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:57:25 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920645, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:57:25.842402 17677 heartbeater.cc:344] Connected to a master server at 127.20.1.188:37811
I20260620 01:57:25.842883 17677 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:25.844381 17677 heartbeater.cc:507] Master 127.20.1.188:37811 requested a full tablet report, sending...
I20260620 01:57:25.846985 17678 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35471
I20260620 01:57:25.847249 17678 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:25.847924 17678 heartbeater.cc:507] Master 127.20.1.190:35471 requested a full tablet report, sending...
I20260620 01:57:25.848201 17471 ts_manager.cc:194] Registered new tserver with Master: 240c1ed18e4b412f8997d62a51aa7b6a (127.20.1.129:41177)
I20260620 01:57:25.850422 17291 ts_manager.cc:194] Registered new tserver with Master: 240c1ed18e4b412f8997d62a51aa7b6a (127.20.1.129:41177)
I20260620 01:57:25.853435 17291 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:33801
I20260620 01:57:25.854453 17681 heartbeater.cc:344] Connected to a master server at 127.20.1.189:37567
I20260620 01:57:25.854660 17681 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.20.1.130@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.130@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:57:25.855386 17681 heartbeater.cc:507] Master 127.20.1.189:37567 requested a full tablet report, sending...
I20260620 01:57:25.857978 17380 ts_manager.cc:194] Registered new tserver with Master: 240c1ed18e4b412f8997d62a51aa7b6a (127.20.1.129:41177)
I20260620 01:57:25.878315 17545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "5e1a5c80089949c18d5b0336c188a008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:25.878674 17545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.130 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.130 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:57:25.895746 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.130: real 0.101s user 0.002s sys 0.024s
WARNING: no policy specified for HTTP/127.20.1.130@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.130@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.130 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.130 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:57:25.971249 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.130:0
--local_ip_for_outbound_sockets=127.20.1.130
--webserver_interface=127.20.1.130
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:57:26.502537 17694 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:26.503042 17694 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:26.503260 17694 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:26.514676 17694 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:26.514930 17694 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.130
I20260620 01:57:26.531267 17694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.1.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.130
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:26.541530 17694 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:26.543493 17694 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:26.562956 17699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:26.562983 17700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:26.563627 17702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:26.570672 17694 server_base.cc:1061] running on GCE node
I20260620 01:57:26.571645 17694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:26.575469 17694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:26.579465 17694 hybrid_clock.cc:648] HybridClock initialized: now 1781920646579150 us; error 220 us; skew 500 ppm
I20260620 01:57:26.581223 17694 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:57:26 dist-test-slave-wg14 krb5kdc[17054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920646, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:57:26.586935 17694 init.cc:377] Logged in from keytab as kudu/127.20.1.130@KRBTEST.COM (short username kudu)
I20260620 01:57:26.589952 17694 webserver.cc:533] Webserver started at http://127.20.1.130:41703/ using document root <none> and password file <none>
I20260620 01:57:26.591033 17694 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:26.591256 17694 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:26.606137 17694 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:57:26.610502 17709 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:26.613734 17694 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260620 01:57:26.614775 17694 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/wal
uuid: "e4c38bccb47c4d139947f898d3ffa7c2"
format_stamp: "Formatted at 2026-06-20 01:57:02 on dist-test-slave-wg14"
I20260620 01:57:26.617388 17694 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:26.663923 17694 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:26.677302 17694 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:26.677788 17694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:26.682168 17694 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:26.684710 17694 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:26.684878 17694 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:26.685019 17694 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:26.685114 17694 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:26.747937 17694 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.130:43325
I20260620 01:57:26.748080 17824 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.130:43325 every 8 connection(s)
I20260620 01:57:26.749831 17694 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/data/info.pb
I20260620 01:57:26.753973 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 17694
I20260620 01:57:26.754271 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-1/wal/instance
Jun 20 01:57:26 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920646, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:57:26 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920646, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:57:26 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920646, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.130@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.131@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.20.1.131@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260620 01:57:26.823256 17827 heartbeater.cc:344] Connected to a master server at 127.20.1.189:37567
I20260620 01:57:26.823868 17827 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:26.825250 17827 heartbeater.cc:507] Master 127.20.1.189:37567 requested a full tablet report, sending...
I20260620 01:57:26.826467 17826 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35471
I20260620 01:57:26.826704 17826 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:26.827411 17826 heartbeater.cc:507] Master 127.20.1.190:35471 requested a full tablet report, sending...
I20260620 01:57:26.823030 17825 heartbeater.cc:344] Connected to a master server at 127.20.1.188:37811
I20260620 01:57:26.828145 17825 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:26.828781 17825 heartbeater.cc:507] Master 127.20.1.188:37811 requested a full tablet report, sending...
I20260620 01:57:26.829346 17291 ts_manager.cc:194] Registered new tserver with Master: e4c38bccb47c4d139947f898d3ffa7c2 (127.20.1.130:43325)
I20260620 01:57:26.830550 17471 ts_manager.cc:194] Registered new tserver with Master: e4c38bccb47c4d139947f898d3ffa7c2 (127.20.1.130:43325)
I20260620 01:57:26.830576 17380 ts_manager.cc:194] Registered new tserver with Master: e4c38bccb47c4d139947f898d3ffa7c2 (127.20.1.130:43325)
I20260620 01:57:26.830824 17291 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:39575
I20260620 01:57:26.857764 17678 heartbeater.cc:499] Master 127.20.1.190:35471 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.131 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.131 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:57:26.873188 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.131: real 0.111s user 0.000s sys 0.030s
WARNING: no policy specified for HTTP/127.20.1.131@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.1.131@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.131 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.131 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:57:26.951480 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.131:0
--local_ip_for_outbound_sockets=127.20.1.131
--webserver_interface=127.20.1.131
--webserver_port=0
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:57:27.582868 17839 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:27.583356 17839 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:27.583501 17839 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:27.602617 17839 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:27.602885 17839 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.131
I20260620 01:57:27.621730 17839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.1.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.131
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:27.624019 17839 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:27.626029 17839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:27.643566 17845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:27.644160 17846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:27.646317 17848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:27.646407 17839 server_base.cc:1061] running on GCE node
I20260620 01:57:27.647337 17839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:27.648994 17839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:27.652735 17839 hybrid_clock.cc:648] HybridClock initialized: now 1781920647652357 us; error 121 us; skew 500 ppm
I20260620 01:57:27.653542 17839 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:57:27 dist-test-slave-wg14 krb5kdc[17054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920647, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:57:27.662528 17839 init.cc:377] Logged in from keytab as kudu/127.20.1.131@KRBTEST.COM (short username kudu)
I20260620 01:57:27.665326 17839 webserver.cc:533] Webserver started at http://127.20.1.131:35393/ using document root <none> and password file <none>
I20260620 01:57:27.666292 17839 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:27.666483 17839 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:27.677372 17839 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.006s sys 0.001s
I20260620 01:57:27.682097 17856 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:27.684129 17839 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:57:27.684377 17839 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/wal
uuid: "a8fc2dc1fbaa420eaadc512eda60b0db"
format_stamp: "Formatted at 2026-06-20 01:57:03 on dist-test-slave-wg14"
I20260620 01:57:27.685392 17839 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:27.724193 17839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:27.730352 17839 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:27.730671 17839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:27.733704 17839 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:27.740339 17839 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:27.740509 17839 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:27.740641 17839 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:27.740726 17839 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:27.797022 17839 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.131:36013
I20260620 01:57:27.797140 17969 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.131:36013 every 8 connection(s)
I20260620 01:57:27.799734 17839 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/data/info.pb
I20260620 01:57:27.800717 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 17839
I20260620 01:57:27.801111 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-2/wal/instance
Jun 20 01:57:27 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920647, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:57:27 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920647, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:57:27 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920647, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.131@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:57:27.836015 17826 heartbeater.cc:499] Master 127.20.1.190:35471 was elected leader, sending a full tablet report...
I20260620 01:57:27.844671 17972 heartbeater.cc:344] Connected to a master server at 127.20.1.189:37567
I20260620 01:57:27.844777 17971 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35471
I20260620 01:57:27.845330 17972 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:27.845326 17971 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:27.846732 17970 heartbeater.cc:344] Connected to a master server at 127.20.1.188:37811
I20260620 01:57:27.847004 17970 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:27.847244 17972 heartbeater.cc:507] Master 127.20.1.189:37567 requested a full tablet report, sending...
I20260620 01:57:27.847257 17971 heartbeater.cc:507] Master 127.20.1.190:35471 requested a full tablet report, sending...
I20260620 01:57:27.847972 17970 heartbeater.cc:507] Master 127.20.1.188:37811 requested a full tablet report, sending...
I20260620 01:57:27.850155 17290 ts_manager.cc:194] Registered new tserver with Master: a8fc2dc1fbaa420eaadc512eda60b0db (127.20.1.131:36013)
I20260620 01:57:27.850629 17471 ts_manager.cc:194] Registered new tserver with Master: a8fc2dc1fbaa420eaadc512eda60b0db (127.20.1.131:36013)
I20260620 01:57:27.851085 17380 ts_manager.cc:194] Registered new tserver with Master: a8fc2dc1fbaa420eaadc512eda60b0db (127.20.1.131:36013)
I20260620 01:57:27.854662 17290 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:57041
I20260620 01:57:27.868808 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:57:27.886387 20486 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260620 01:57:27.886636 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 17258
I20260620 01:57:27.888329 17321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:28.029731 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 17258
W20260620 01:57:28.050275 17458 connection.cc:573] server connection from 127.0.0.1:34204 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:57:28.051317 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:35471
--webserver_interface=127.20.1.190
--webserver_port=42791
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:57:28.523844 17981 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:28.524336 17981 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:28.524511 17981 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:28.535784 17981 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:28.536008 17981 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:28.536095 17981 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:28.536172 17981 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:28.551492 17981 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:35471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=42791
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:28.554347 17981 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:28.558655 17981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:28.573577 17986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:28.573799 17987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:28.574752 17989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:28.576220 17981 server_base.cc:1061] running on GCE node
I20260620 01:57:28.577517 17981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:28.581532 17981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:28.585212 17981 hybrid_clock.cc:648] HybridClock initialized: now 1781920648584995 us; error 111 us; skew 500 ppm
I20260620 01:57:28.585915 17981 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:57:28 dist-test-slave-wg14 krb5kdc[17054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920648, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:57:28.592299 17981 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:57:28.596395 17981 webserver.cc:533] Webserver started at http://127.20.1.190:42791/ using document root <none> and password file <none>
I20260620 01:57:28.597470 17981 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:28.598309 17981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:28.608047 17981 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:57:28.613379 17996 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:28.615688 17981 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:57:28.615993 17981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "5e1a5c80089949c18d5b0336c188a008"
format_stamp: "Formatted at 2026-06-20 01:56:58 on dist-test-slave-wg14"
I20260620 01:57:28.617087 17981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:28.669106 17981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:28.676100 17981 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:28.676407 17981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:28.706610 17981 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:35471
I20260620 01:57:28.706641 18047 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:35471 every 8 connection(s)
I20260620 01:57:28.708788 17981 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:57:28.712880 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 17981
I20260620 01:57:28.713281 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 17540
I20260620 01:57:28.719779 18048 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:57:28.725234 18048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Bootstrap starting.
Jun 20 01:57:28 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 01:57:28.743731 17672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:28.751960 18048 log.cc:826] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:28.765798 18048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:57:28.766659 18048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: Bootstrap complete.
I20260620 01:57:28.775774 18048 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:28.776791 18048 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1a5c80089949c18d5b0336c188a008, State: Initialized, Role: FOLLOWER
I20260620 01:57:28.777763 18048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:28.778823 18054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:28.779354 18054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:28.780649 18048 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:28.803107 18065 catalog_manager.cc:1294] Loaded cluster ID: c67ebe9c0bd849748e5f573512cf92ac
I20260620 01:57:28.803360 18065 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: loading cluster ID for follower catalog manager: success
I20260620 01:57:28.808548 18065 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: acquiring CA information for follower catalog manager: success
I20260620 01:57:28.815264 18065 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:57:28.860561 17971 heartbeater.cc:499] Master 127.20.1.190:35471 was elected leader, sending a full tablet report...
I20260620 01:57:28.889879 18013 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e4c38bccb47c4d139947f898d3ffa7c2" instance_seqno: 1781920646729477) as {username='kudu', principal='kudu/127.20.1.130@KRBTEST.COM'} at 127.20.1.130:53167; Asking this server to re-register.
I20260620 01:57:28.893443 17826 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:28.896754 18013 ts_manager.cc:194] Registered new tserver with Master: e4c38bccb47c4d139947f898d3ffa7c2 (127.20.1.130:43325)
I20260620 01:57:28.917284 17981 master_runner.cc:264] No leader master found from master 5e1a5c80089949c18d5b0336c188a008
I20260620 01:57:28.917498 17981 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:57:28.921237 18013 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "a8fc2dc1fbaa420eaadc512eda60b0db" instance_seqno: 1781920647786070) as {username='kudu', principal='kudu/127.20.1.131@KRBTEST.COM'} at 127.20.1.131:51337; Asking this server to re-register.
I20260620 01:57:28.922688 17971 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:28.925213 18013 ts_manager.cc:194] Registered new tserver with Master: a8fc2dc1fbaa420eaadc512eda60b0db (127.20.1.131:36013)
I20260620 01:57:28.936790 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 17540
W20260620 01:57:28.956478 18068 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:57:28.930159 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:37077 (local address 127.20.1.190:35471)
0620 01:57:28.936243 (+ 6084us) server_negotiation.cc:207] Beginning negotiation
0620 01:57:28.936251 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:57:28.954910 (+ 18659us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:37077: BlockingRecv error: recv got EOF from 127.20.1.129:37077 (error 108)
Metrics: {"server-negotiator.queue_time_us":5836}
I20260620 01:57:28.957492 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:41177
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--webserver_port=42223
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:57:29.095517 17348 master_runner.cc:264] No leader master found from master 7a004ebbc3c54fbe98351e428a62af87
I20260620 01:57:29.095717 17348 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:57:29.282074 18078 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5e1a5c80089949c18d5b0336c188a008)
I20260620 01:57:29.282424 18078 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:29.285917 18078 leader_election.cc:290] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e1a5c80089949c18d5b0336c188a008 (127.20.1.190:35471), 9b772652276745cabab57b0bd8ef5983 (127.20.1.188:37811)
I20260620 01:57:29.287091 17480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a004ebbc3c54fbe98351e428a62af87" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9b772652276745cabab57b0bd8ef5983" is_pre_election: true
I20260620 01:57:29.287432 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a004ebbc3c54fbe98351e428a62af87" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e1a5c80089949c18d5b0336c188a008" is_pre_election: true
I20260620 01:57:29.288168 18023 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a004ebbc3c54fbe98351e428a62af87 in term 3.
I20260620 01:57:29.289435 17364 leader_election.cc:304] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e1a5c80089949c18d5b0336c188a008, 7a004ebbc3c54fbe98351e428a62af87; no voters:
I20260620 01:57:29.290195 18078 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 01:57:29.290464 18078 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5e1a5c80089949c18d5b0336c188a008)
I20260620 01:57:29.290593 18078 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 3 FOLLOWER]: Advancing to term 4
I20260620 01:57:29.294466 18078 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:29.296037 18078 leader_election.cc:290] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [CANDIDATE]: Term 4 election: Requested vote from peers 5e1a5c80089949c18d5b0336c188a008 (127.20.1.190:35471), 9b772652276745cabab57b0bd8ef5983 (127.20.1.188:37811)
I20260620 01:57:29.296800 17480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a004ebbc3c54fbe98351e428a62af87" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9b772652276745cabab57b0bd8ef5983"
I20260620 01:57:29.296942 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a004ebbc3c54fbe98351e428a62af87" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e1a5c80089949c18d5b0336c188a008"
I20260620 01:57:29.297243 18023 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 3 FOLLOWER]: Advancing to term 4
I20260620 01:57:29.304840 18023 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a004ebbc3c54fbe98351e428a62af87 in term 4.
I20260620 01:57:29.305924 17364 leader_election.cc:304] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e1a5c80089949c18d5b0336c188a008, 7a004ebbc3c54fbe98351e428a62af87; no voters: 9b772652276745cabab57b0bd8ef5983
I20260620 01:57:29.306882 18078 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 01:57:29.307637 18078 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [term 4 LEADER]: Becoming Leader. State: Replica: 7a004ebbc3c54fbe98351e428a62af87, State: Running, Role: LEADER
I20260620 01:57:29.308409 18078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:29.311358 18079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a004ebbc3c54fbe98351e428a62af87. Latest consensus state: current_term: 4 leader_uuid: "7a004ebbc3c54fbe98351e428a62af87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:29.311961 18079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:29.313107 18081 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:57:29.316504 18081 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:57:29.317919 18081 catalog_manager.cc:1294] Loaded cluster ID: c67ebe9c0bd849748e5f573512cf92ac
I20260620 01:57:29.318077 18081 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:57:29.320107 18081 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:57:29.322027 18081 catalog_manager.cc:6080] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87: Loaded TSK: 0
I20260620 01:57:29.323817 18081 catalog_manager.cc:1549] Initializing in-progress tserver states...
W20260620 01:57:29.473968 18076 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:29.474421 18076 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:29.474586 18076 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:29.485499 18076 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:29.485823 18076 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:29.502461 18076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:41177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=42223
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:29.505568 18076 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:29.507530 18076 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:29.524396 18086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:29.524554 18089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:29.524518 18087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:29.526054 18076 server_base.cc:1061] running on GCE node
I20260620 01:57:29.527359 18076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:29.529561 18076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:29.531477 18092 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5e1a5c80089949c18d5b0336c188a008)
I20260620 01:57:29.531785 18092 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } }
I20260620 01:57:29.533166 18076 hybrid_clock.cc:648] HybridClock initialized: now 1781920649532910 us; error 141 us; skew 500 ppm
I20260620 01:57:29.533835 18076 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:29.534909 18092 leader_election.cc:290] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e1a5c80089949c18d5b0336c188a008 (127.20.1.190:35471), 7a004ebbc3c54fbe98351e428a62af87 (127.20.1.189:37567)
I20260620 01:57:29.536326 17390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b772652276745cabab57b0bd8ef5983" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a004ebbc3c54fbe98351e428a62af87" is_pre_election: true
Jun 20 01:57:29 dist-test-slave-wg14 krb5kdc[17054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920649, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:57:29.539937 18076 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:57:29.544463 18076 webserver.cc:533] Webserver started at http://127.20.1.129:42223/ using document root <none> and password file <none>
I20260620 01:57:29.544818 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b772652276745cabab57b0bd8ef5983" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e1a5c80089949c18d5b0336c188a008" is_pre_election: true
I20260620 01:57:29.545466 18023 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9b772652276745cabab57b0bd8ef5983 in current term 4: Already voted for candidate 7a004ebbc3c54fbe98351e428a62af87 in this term.
I20260620 01:57:29.545748 18076 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:29.545904 18076 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:29.546726 17455 leader_election.cc:304] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b772652276745cabab57b0bd8ef5983; no voters: 5e1a5c80089949c18d5b0336c188a008, 7a004ebbc3c54fbe98351e428a62af87
I20260620 01:57:29.547878 18092 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 3 FOLLOWER]: Advancing to term 4
I20260620 01:57:29.550431 18092 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 01:57:29.556382 18076 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 01:57:29.561691 18098 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:29.564733 18076 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.004s sys 0.001s
I20260620 01:57:29.565738 18076 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "240c1ed18e4b412f8997d62a51aa7b6a"
format_stamp: "Formatted at 2026-06-20 01:57:01 on dist-test-slave-wg14"
I20260620 01:57:29.567343 18076 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:29.591962 18076 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:29.601951 18076 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:29.602335 18076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:29.603816 18076 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:29.606206 18076 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:29.606371 18076 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:29.606519 18076 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:29.606606 18076 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:29.672879 18076 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:41177
I20260620 01:57:29.672969 18211 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:41177 every 8 connection(s)
I20260620 01:57:29.674974 18076 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:57:29.684419 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 18076
Jun 20 01:57:29 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Jun 20 01:57:29 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920649, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:57:29 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920649, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:57:29 dist-test-slave-wg14 krb5kdc[17054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920649, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:57:29.743844 18212 heartbeater.cc:344] Connected to a master server at 127.20.1.188:37811
I20260620 01:57:29.744516 18212 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:29.746131 18212 heartbeater.cc:507] Master 127.20.1.188:37811 requested a full tablet report, sending...
I20260620 01:57:29.749152 18213 heartbeater.cc:344] Connected to a master server at 127.20.1.190:35471
I20260620 01:57:29.749380 18213 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:29.750909 18213 heartbeater.cc:507] Master 127.20.1.190:35471 requested a full tablet report, sending...
I20260620 01:57:29.751896 17471 ts_manager.cc:194] Re-registered known tserver with Master: 240c1ed18e4b412f8997d62a51aa7b6a (127.20.1.129:41177)
I20260620 01:57:29.755584 18013 ts_manager.cc:194] Registered new tserver with Master: 240c1ed18e4b412f8997d62a51aa7b6a (127.20.1.129:41177)
I20260620 01:57:29.761217 17480 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [term 4 FOLLOWER]: Refusing update from remote peer 7a004ebbc3c54fbe98351e428a62af87: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260620 01:57:29.761269 18214 heartbeater.cc:344] Connected to a master server at 127.20.1.189:37567
I20260620 01:57:29.761618 18214 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:29.762639 18214 heartbeater.cc:507] Master 127.20.1.189:37567 requested a full tablet report, sending...
I20260620 01:57:29.763495 18079 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 01:57:29.764827 17380 ts_manager.cc:194] Re-registered known tserver with Master: 240c1ed18e4b412f8997d62a51aa7b6a (127.20.1.129:41177)
I20260620 01:57:29.767524 18092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a004ebbc3c54fbe98351e428a62af87. Latest consensus state: current_term: 4 leader_uuid: "7a004ebbc3c54fbe98351e428a62af87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:29.768070 18092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:29.767987 17380 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:35187
I20260620 01:57:29.783354 18092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "7a004ebbc3c54fbe98351e428a62af87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:29.783825 18092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b772652276745cabab57b0bd8ef5983 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:29.784596 18079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "7a004ebbc3c54fbe98351e428a62af87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:29.785260 18079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:29.786288 18023 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [term 4 FOLLOWER]: Refusing update from remote peer 7a004ebbc3c54fbe98351e428a62af87: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260620 01:57:29.787731 18079 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 01:57:29.805902 18229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "7a004ebbc3c54fbe98351e428a62af87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:29.806461 18229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:29.806974 18226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a004ebbc3c54fbe98351e428a62af87. Latest consensus state: current_term: 4 leader_uuid: "7a004ebbc3c54fbe98351e428a62af87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:29.807389 18226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1a5c80089949c18d5b0336c188a008 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:29.944070 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 18076
W20260620 01:57:29.947634 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 18076: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:41177: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:57:29.947861 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 18076
I20260620 01:57:29.966210 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:41177
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--webserver_port=42223
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:57:30.169512 18225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "7a004ebbc3c54fbe98351e428a62af87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1a5c80089949c18d5b0336c188a008" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 35471 } } peers { permanent_uuid: "7a004ebbc3c54fbe98351e428a62af87" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 37567 } } peers { permanent_uuid: "9b772652276745cabab57b0bd8ef5983" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37811 } } }
I20260620 01:57:30.169943 18225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a004ebbc3c54fbe98351e428a62af87 [sys.catalog]: This master's current role is: LEADER
W20260620 01:57:30.518323 18235 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:30.518806 18235 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:30.518918 18235 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:57:30.518970 18235 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:30.530062 18235 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:30.530362 18235 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:30.545069 18235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:41177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=42223
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:30.547336 18235 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:30.549330 18235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:30.565564 18241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:30.565573 18240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:30.566737 18243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:30.569169 18235 server_base.cc:1061] running on GCE node
I20260620 01:57:30.570044 18235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:30.572438 18235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:30.580801 18235 hybrid_clock.cc:648] HybridClock initialized: now 1781920650579975 us; error 503 us; skew 500 ppm
I20260620 01:57:30.581377 18235 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:57:30 dist-test-slave-wg14 krb5kdc[17054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920650, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:57:30.587553 18235 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:57:30.590070 18235 webserver.cc:533] Webserver started at http://127.20.1.129:42223/ using document root <none> and password file <none>
I20260620 01:57:30.591243 18235 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:30.591392 18235 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:30.600696 18235 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 01:57:30.606416 18250 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:30.608814 18235 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:57:30.609077 18235 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "240c1ed18e4b412f8997d62a51aa7b6a"
format_stamp: "Formatted at 2026-06-20 01:57:01 on dist-test-slave-wg14"
I20260620 01:57:30.610086 18235 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:30.645464 18235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:30.651119 18235 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:30.651499 18235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:30.653844 18235 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:30.657558 18235 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:30.657917 18235 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:30.658135 18235 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:30.658179 18235 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:57:30.662034 18235 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:57:30.662209 18235 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:57:30.662314 18235 fault_injection.cc:59] injecting fault for kudu (pid 18235): FLAGS_fault_before_start (process will exit)
I20260620 01:57:30.683516 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:41177
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--webserver_port=42223
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:57:30.845647 17827 heartbeater.cc:499] Master 127.20.1.189:37567 was elected leader, sending a full tablet report...
I20260620 01:57:30.867249 17972 heartbeater.cc:499] Master 127.20.1.189:37567 was elected leader, sending a full tablet report...
W20260620 01:57:31.222450 18258 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:31.222913 18258 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:31.223099 18258 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:31.238680 18258 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:31.238946 18258 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:31.256448 18258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:33661,127.20.1.149:42845,127.20.1.150:44095,127.20.1.151:43389,127.20.1.152:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:41177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=42223
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:35471,127.20.1.189:37567,127.20.1.188:37811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:31.258709 18258 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:31.260667 18258 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:31.277732 18263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:31.277710 18264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:31.279436 18266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:31.281535 18258 server_base.cc:1061] running on GCE node
I20260620 01:57:31.282601 18258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:31.284368 18258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:31.287930 18258 hybrid_clock.cc:648] HybridClock initialized: now 1781920651287760 us; error 69 us; skew 500 ppm
I20260620 01:57:31.288488 18258 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:57:31 dist-test-slave-wg14 krb5kdc[17054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920651, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:57:31.293772 18258 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:57:31.296401 18258 webserver.cc:533] Webserver started at http://127.20.1.129:42223/ using document root <none> and password file <none>
I20260620 01:57:31.297415 18258 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:31.297597 18258 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:31.305507 18258 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:57:31.310528 18273 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:31.312744 18258 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:57:31.313010 18258 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "240c1ed18e4b412f8997d62a51aa7b6a"
format_stamp: "Formatted at 2026-06-20 01:57:01 on dist-test-slave-wg14"
I20260620 01:57:31.314427 18258 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:31.360705 18258 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:31.366583 18258 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:31.366935 18258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:31.368280 18258 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:31.370565 18258 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:31.370762 18258 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:31.370914 18258 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:31.370995 18258 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:31.429255 18258 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:41177
I20260620 01:57:31.429324 18386 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:41177 every 8 connection(s)
I20260620 01:57:31.431360 18258 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:57:31.434057 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 18258
I20260620 01:57:31.434497 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 18258
W20260620 01:57:31.450148 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 18258: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:41177: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:57:31.450350 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 18258
W20260620 01:57:31.468223 18393 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:57:31.436830 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:33333 (local address 127.20.1.190:35471)
0620 01:57:31.438329 (+ 1499us) server_negotiation.cc:207] Beginning negotiation
0620 01:57:31.438336 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 01:57:31.444110 (+ 5774us) server_negotiation.cc:408] Connection header received
0620 01:57:31.448221 (+ 4111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:57:31.448230 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:57:31.448334 (+ 104us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:57:31.448443 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:57:31.467289 (+ 18846us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:33333: BlockingRecv error: recv got EOF from 127.20.1.129:33333 (error 108)
Metrics: {"server-negotiator.queue_time_us":1370,"thread_start_us":1477,"threads_started":1}
W20260620 01:57:31.468492 18394 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:57:31.438701 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:35847 (local address 127.20.1.188:37811)
0620 01:57:31.442730 (+ 4029us) server_negotiation.cc:207] Beginning negotiation
0620 01:57:31.442745 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0620 01:57:31.446364 (+ 3619us) server_negotiation.cc:408] Connection header received
0620 01:57:31.447572 (+ 1208us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:57:31.447580 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:57:31.447691 (+ 111us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:57:31.447826 (+ 135us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:57:31.467153 (+ 19327us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:35847: BlockingRecv error: recv got EOF from 127.20.1.129:35847 (error 108)
Metrics: {"server-negotiator.queue_time_us":3585,"thread_start_us":3313,"threads_started":1}
W20260620 01:57:31.468473 18392 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:57:31.436204 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:33571 (local address 127.20.1.189:37567)
0620 01:57:31.436840 (+ 636us) server_negotiation.cc:207] Beginning negotiation
0620 01:57:31.436847 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 01:57:31.444199 (+ 7352us) server_negotiation.cc:408] Connection header received
0620 01:57:31.447618 (+ 3419us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:57:31.447627 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:57:31.447745 (+ 118us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:57:31.447872 (+ 127us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:57:31.467149 (+ 19277us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:33571: BlockingRecv error: recv got EOF from 127.20.1.129:33571 (error 108)
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":360,"threads_started":1}
I20260620 01:57:31.469121 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 17694
W20260620 01:57:31.474629 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 17694: Not authorized: Client connection negotiation failed: client connection to 127.20.1.130:43325: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:57:31.474892 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 17694
I20260620 01:57:31.490818 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 17839
W20260620 01:57:31.495352 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 17839: Not authorized: Client connection negotiation failed: client connection to 127.20.1.131:36013: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:57:31.495703 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 17839
I20260620 01:57:31.513857 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 17981
W20260620 01:57:31.516857 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 17981: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:35471: server requires authentication, but client does not have Kerberos credentials available
I20260620 01:57:31.517181 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 17981
W20260620 01:57:31.517215 18393 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:57:31.515084 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.20.1.190:35471)
0620 01:57:31.515386 (+ 302us) server_negotiation.cc:207] Beginning negotiation
0620 01:57:31.515393 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 01:57:31.515420 (+ 27us) server_negotiation.cc:408] Connection header received
0620 01:57:31.515511 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:57:31.515517 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:57:31.515610 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:57:31.515753 (+ 143us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:57:31.516940 (+ 1187us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57722: BlockingRecv error: recv got EOF from 127.0.0.1:57722 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 01:57:31.541656 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 17348
I20260620 01:57:31.543443 17410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:31.711356 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 17348
I20260620 01:57:31.731813 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 17438
I20260620 01:57:31.734062 17501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:31.883100 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 17438
Shutting down hive metastore.
2026-06-20T01:57:32Z chronyd exiting
2026-06-20T01:57:32Z chronyd exiting
2026-06-20T01:57:32Z chronyd exiting
2026-06-20T01:57:32Z chronyd exiting
2026-06-20T01:57:32Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/11 (58905 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/0
I20260620 01:57:32.395097 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44241
--webserver_interface=127.20.1.190
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661 with env {}
W20260620 01:57:33.017789 18409 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:33.019059 18409 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:33.019783 18409 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:33.031801 18409 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:33.031972 18409 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:33.032223 18409 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:33.032578 18409 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:33.052709 18409 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:33.054778 18409 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:33.057147 18409 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:33.072803 18415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:33.072860 18417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:33.073287 18414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:33.074352 18409 server_base.cc:1061] running on GCE node
I20260620 01:57:33.075881 18409 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:57:33.076455 18409 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:57:33.076637 18409 hybrid_clock.cc:648] HybridClock initialized: now 1781920653076596 us; error 0 us; skew 500 ppm
I20260620 01:57:33.077338 18409 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:33.083941 18409 webserver.cc:533] Webserver started at http://127.20.1.190:40113/ using document root <none> and password file <none>
I20260620 01:57:33.085129 18409 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:33.085300 18409 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:33.085821 18409 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:33.089596 18409 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "10dc7a82a39746cbbe804607533b5499"
format_stamp: "Formatted at 2026-06-20 01:57:33 on dist-test-slave-wg14"
I20260620 01:57:33.090776 18409 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "10dc7a82a39746cbbe804607533b5499"
format_stamp: "Formatted at 2026-06-20 01:57:33 on dist-test-slave-wg14"
I20260620 01:57:33.101732 18409 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.000s sys 0.008s
I20260620 01:57:33.107523 18422 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:33.110369 18409 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.001s sys 0.002s
I20260620 01:57:33.110632 18409 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "10dc7a82a39746cbbe804607533b5499"
format_stamp: "Formatted at 2026-06-20 01:57:33 on dist-test-slave-wg14"
I20260620 01:57:33.110942 18409 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:33.154997 18409 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:33.156287 18409 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:33.156629 18409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:33.197736 18473 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44241 every 8 connection(s)
I20260620 01:57:33.197726 18409 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44241
I20260620 01:57:33.200455 18409 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:57:33.204348 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 18409
I20260620 01:57:33.204610 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:57:33.206360 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:35113
--webserver_interface=127.20.1.189
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661 with env {}
I20260620 01:57:33.208461 18474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:57:33.223825 18474 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:33.269054 18474 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:57:33.270902 18423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:35113: connect: Connection refused (error 111)
W20260620 01:57:33.278494 18474 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:35113: Network error: Client connection negotiation failed: client connection to 127.20.1.189:35113: connect: Connection refused (error 111)
I20260620 01:57:33.328091 18474 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } attempt: 1
W20260620 01:57:33.333755 18474 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:35113: Network error: Client connection negotiation failed: client connection to 127.20.1.189:35113: connect: Connection refused (error 111)
I20260620 01:57:33.402303 18474 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } attempt: 2
W20260620 01:57:33.404744 18474 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:35113: Network error: Client connection negotiation failed: client connection to 127.20.1.189:35113: connect: Connection refused (error 111)
I20260620 01:57:33.496217 18474 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } attempt: 3
W20260620 01:57:33.499432 18474 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:35113: Network error: Client connection negotiation failed: client connection to 127.20.1.189:35113: connect: Connection refused (error 111)
I20260620 01:57:33.642907 18474 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } attempt: 4
W20260620 01:57:33.646502 18474 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:35113: Network error: Client connection negotiation failed: client connection to 127.20.1.189:35113: connect: Connection refused (error 111)
W20260620 01:57:33.732376 18476 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:33.732932 18476 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:33.733052 18476 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:33.743083 18476 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:33.743283 18476 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:33.743358 18476 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:33.743402 18476 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:33.760820 18476 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:35113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:33.763119 18476 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:33.766944 18476 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:33.784175 18484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:33.785202 18486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:33.787117 18488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:33.788079 18476 server_base.cc:1061] running on GCE node
I20260620 01:57:33.789876 18476 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:57:33.790434 18476 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:57:33.790588 18476 hybrid_clock.cc:648] HybridClock initialized: now 1781920653790565 us; error 0 us; skew 500 ppm
I20260620 01:57:33.791217 18476 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:33.796042 18476 webserver.cc:533] Webserver started at http://127.20.1.189:42657/ using document root <none> and password file <none>
I20260620 01:57:33.797096 18476 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:33.797214 18476 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:33.797569 18476 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:33.800796 18476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "fffc8f7d4da14530893ed18893e13ed5"
format_stamp: "Formatted at 2026-06-20 01:57:33 on dist-test-slave-wg14"
I20260620 01:57:33.801870 18476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "fffc8f7d4da14530893ed18893e13ed5"
format_stamp: "Formatted at 2026-06-20 01:57:33 on dist-test-slave-wg14"
I20260620 01:57:33.811728 18476 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.011s sys 0.000s
I20260620 01:57:33.819409 18493 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:33.821864 18476 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260620 01:57:33.822125 18476 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "fffc8f7d4da14530893ed18893e13ed5"
format_stamp: "Formatted at 2026-06-20 01:57:33 on dist-test-slave-wg14"
I20260620 01:57:33.822642 18476 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:33.850512 18476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:33.852632 18476 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:33.853286 18476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:33.883291 18476 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:35113
I20260620 01:57:33.883316 18544 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:35113 every 8 connection(s)
I20260620 01:57:33.886675 18476 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:57:33.891316 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 18476
I20260620 01:57:33.891598 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:57:33.892982 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:40661
--webserver_interface=127.20.1.188
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661 with env {}
I20260620 01:57:33.893517 18545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:57:33.907188 18545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:33.930521 18545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:33.943760 18545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:33.945909 18474 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } attempt: 5
W20260620 01:57:33.946036 18496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40661: connect: Connection refused (error 111)
W20260620 01:57:33.948675 18545 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40661: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40661: connect: Connection refused (error 111)
I20260620 01:57:33.962253 18474 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:57:33.966727 18474 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40661: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40661: connect: Connection refused (error 111)
I20260620 01:57:33.998176 18545 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } attempt: 1
W20260620 01:57:34.001806 18545 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40661: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40661: connect: Connection refused (error 111)
I20260620 01:57:34.018177 18474 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } attempt: 1
W20260620 01:57:34.022435 18474 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40661: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40661: connect: Connection refused (error 111)
I20260620 01:57:34.070356 18545 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } attempt: 2
W20260620 01:57:34.073148 18545 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40661: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40661: connect: Connection refused (error 111)
I20260620 01:57:34.090754 18474 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } attempt: 2
W20260620 01:57:34.096642 18474 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40661: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40661: connect: Connection refused (error 111)
I20260620 01:57:34.165155 18545 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } attempt: 3
W20260620 01:57:34.168856 18545 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40661: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40661: connect: Connection refused (error 111)
I20260620 01:57:34.203110 18474 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } attempt: 3
W20260620 01:57:34.206997 18474 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40661: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40661: connect: Connection refused (error 111)
I20260620 01:57:34.312217 18545 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } attempt: 4
W20260620 01:57:34.315514 18545 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40661: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40661: connect: Connection refused (error 111)
I20260620 01:57:34.384387 18474 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } attempt: 4
W20260620 01:57:34.388060 18474 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40661: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40661: connect: Connection refused (error 111)
W20260620 01:57:34.512826 18547 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:34.513273 18547 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:34.513411 18547 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:34.525158 18547 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:34.525372 18547 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:34.525745 18547 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:34.525929 18547 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:34.542271 18547 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:40661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:34.544850 18547 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:34.549252 18547 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:34.563373 18557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:34.563333 18556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:34.565455 18559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:34.567257 18547 server_base.cc:1061] running on GCE node
I20260620 01:57:34.568423 18547 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:57:34.568693 18547 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:57:34.568847 18547 hybrid_clock.cc:648] HybridClock initialized: now 1781920654568821 us; error 0 us; skew 500 ppm
I20260620 01:57:34.569316 18547 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:34.574604 18547 webserver.cc:533] Webserver started at http://127.20.1.188:44047/ using document root <none> and password file <none>
I20260620 01:57:34.578052 18547 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:34.578269 18547 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:34.581671 18547 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:34.586102 18547 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "3b2d31e78d1c41fabe77181adc16a266"
format_stamp: "Formatted at 2026-06-20 01:57:34 on dist-test-slave-wg14"
I20260620 01:57:34.587360 18547 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "3b2d31e78d1c41fabe77181adc16a266"
format_stamp: "Formatted at 2026-06-20 01:57:34 on dist-test-slave-wg14"
I20260620 01:57:34.602373 18547 fs_manager.cc:696] Time spent creating directory manager: real 0.015s user 0.005s sys 0.008s
I20260620 01:57:34.607997 18564 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:34.610558 18547 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260620 01:57:34.610865 18547 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "3b2d31e78d1c41fabe77181adc16a266"
format_stamp: "Formatted at 2026-06-20 01:57:34 on dist-test-slave-wg14"
I20260620 01:57:34.611246 18547 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:34.615041 18545 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } attempt: 5
W20260620 01:57:34.618744 18545 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40661: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40661: connect: Connection refused (error 111)
I20260620 01:57:34.664548 18547 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:34.665372 18474 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } attempt: 5
I20260620 01:57:34.665895 18547 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:34.666209 18547 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260620 01:57:34.668838 18474 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40661: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40661: connect: Connection refused (error 111)
I20260620 01:57:34.708216 18547 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:40661
I20260620 01:57:34.708261 18615 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:40661 every 8 connection(s)
I20260620 01:57:34.710830 18547 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:57:34.717443 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 18547
I20260620 01:57:34.718945 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:57:34.721464 18616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:57:34.738763 18616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:34.764529 18616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:34.782658 18616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:34.798369 18616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266: Bootstrap starting.
I20260620 01:57:34.802942 18616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266: Neither blocks nor log segments found. Creating new log.
I20260620 01:57:34.805783 18616 log.cc:826] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:34.811817 18616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266: No bootstrap required, opened a new log
I20260620 01:57:34.822991 18616 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } }
I20260620 01:57:34.823527 18616 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:57:34.823686 18616 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b2d31e78d1c41fabe77181adc16a266, State: Initialized, Role: FOLLOWER
I20260620 01:57:34.824550 18616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } }
I20260620 01:57:34.825915 18624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } }
I20260620 01:57:34.826287 18624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:34.827559 18616 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:57:34.844877 18635 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:57:34.845144 18635 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:57:34.840479 18547 master_runner.cc:187] Error getting master registration for 127.20.1.190:44241: OK, instance_id { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" instance_seqno: 1781920653186634 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:57:34.848927 18547 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:57:34.922603 18570 tablet.cc:2406] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:57:35.166215 18545 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } attempt: 6
I20260620 01:57:35.187290 18545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5: Bootstrap starting.
I20260620 01:57:35.191494 18545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5: Neither blocks nor log segments found. Creating new log.
I20260620 01:57:35.193699 18474 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } attempt: 6
I20260620 01:57:35.195528 18545 log.cc:826] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:35.211591 18624 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:57:35.212498 18624 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } }
I20260620 01:57:35.228582 18624 leader_election.cc:290] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10dc7a82a39746cbbe804607533b5499 (127.20.1.190:44241), fffc8f7d4da14530893ed18893e13ed5 (127.20.1.189:35113)
I20260620 01:57:35.249558 18545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5: No bootstrap required, opened a new log
I20260620 01:57:35.254096 18474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499: Bootstrap starting.
I20260620 01:57:35.264716 18545 raft_consensus.cc:359] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } }
I20260620 01:57:35.265558 18545 raft_consensus.cc:385] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:57:35.265798 18545 raft_consensus.cc:740] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fffc8f7d4da14530893ed18893e13ed5, State: Initialized, Role: FOLLOWER
I20260620 01:57:35.266511 18474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499: Neither blocks nor log segments found. Creating new log.
I20260620 01:57:35.267969 18545 consensus_queue.cc:260] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } }
I20260620 01:57:35.268349 18474 log.cc:826] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:35.275344 18545 sys_catalog.cc:565] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:35.283102 18474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499: No bootstrap required, opened a new log
I20260620 01:57:35.284760 18520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b2d31e78d1c41fabe77181adc16a266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fffc8f7d4da14530893ed18893e13ed5" is_pre_election: true
I20260620 01:57:35.286785 18520 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b2d31e78d1c41fabe77181adc16a266 in term 0.
I20260620 01:57:35.289060 18565 leader_election.cc:304] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b2d31e78d1c41fabe77181adc16a266, fffc8f7d4da14530893ed18893e13ed5; no voters:
I20260620 01:57:35.289985 18640 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } }
I20260620 01:57:35.290838 18640 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:35.290544 18624 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:57:35.291715 18624 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:57:35.291915 18624 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:57:35.296025 18624 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } }
I20260620 01:57:35.299471 18624 leader_election.cc:290] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [CANDIDATE]: Term 1 election: Requested vote from peers 10dc7a82a39746cbbe804607533b5499 (127.20.1.190:44241), fffc8f7d4da14530893ed18893e13ed5 (127.20.1.189:35113)
I20260620 01:57:35.300364 18520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b2d31e78d1c41fabe77181adc16a266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fffc8f7d4da14530893ed18893e13ed5"
I20260620 01:57:35.300679 18520 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:57:35.299371 18474 raft_consensus.cc:359] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } }
I20260620 01:57:35.302852 18474 raft_consensus.cc:385] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:57:35.303233 18474 raft_consensus.cc:740] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10dc7a82a39746cbbe804607533b5499, State: Initialized, Role: FOLLOWER
I20260620 01:57:35.304605 18474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } }
I20260620 01:57:35.306298 18520 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b2d31e78d1c41fabe77181adc16a266 in term 1.
I20260620 01:57:35.306344 18642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } }
I20260620 01:57:35.307356 18565 leader_election.cc:304] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b2d31e78d1c41fabe77181adc16a266, fffc8f7d4da14530893ed18893e13ed5; no voters:
I20260620 01:57:35.307387 18642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:35.308091 18624 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:57:35.307940 18448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b2d31e78d1c41fabe77181adc16a266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10dc7a82a39746cbbe804607533b5499"
I20260620 01:57:35.307965 18449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b2d31e78d1c41fabe77181adc16a266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10dc7a82a39746cbbe804607533b5499" is_pre_election: true
I20260620 01:57:35.308609 18449 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b2d31e78d1c41fabe77181adc16a266 in term 0.
I20260620 01:57:35.308987 18474 sys_catalog.cc:565] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:35.311347 18624 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [term 1 LEADER]: Becoming Leader. State: Replica: 3b2d31e78d1c41fabe77181adc16a266, State: Running, Role: LEADER
I20260620 01:57:35.312201 18624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } }
I20260620 01:57:35.326303 18643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b2d31e78d1c41fabe77181adc16a266. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } }
I20260620 01:57:35.328193 18643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:35.343866 18651 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:57:35.348121 18651 catalog_manager.cc:1519] Initializing Kudu cluster ID...
W20260620 01:57:35.362263 18499 tablet.cc:2406] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:57:35.406728 18520 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [term 1 FOLLOWER]: Refusing update from remote peer 3b2d31e78d1c41fabe77181adc16a266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:57:35.409557 18643 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:57:35.410156 18449 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [term 0 FOLLOWER]: Advancing to term 1
W20260620 01:57:35.415813 18428 tablet.cc:2406] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:57:35.416507 18449 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [term 1 FOLLOWER]: Refusing update from remote peer 3b2d31e78d1c41fabe77181adc16a266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:57:35.387702 18476 master_runner.cc:187] Error getting master registration for 127.20.1.190:44241: OK, instance_id { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" instance_seqno: 1781920653186634 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:57:35.417055 18476 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:57:35.418162 18643 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:57:35.436053 18640 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b2d31e78d1c41fabe77181adc16a266. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } }
I20260620 01:57:35.436455 18640 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:35.447414 18640 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } }
I20260620 01:57:35.447716 18640 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:57:35.448436 18665 catalog_manager.cc:1593] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:57:35.448578 18665 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:57:35.454337 18669 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 01:57:35.459064 18643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } }
I20260620 01:57:35.459439 18643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:35.462460 18642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b2d31e78d1c41fabe77181adc16a266. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } }
I20260620 01:57:35.462450 18643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } }
I20260620 01:57:35.462939 18643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:35.462988 18642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:35.471024 18651 catalog_manager.cc:1382] Generated new cluster ID: d2f491a621eb41edb9eb506862560f94
I20260620 01:57:35.471233 18651 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:57:35.474812 18642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } }
I20260620 01:57:35.475353 18642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:35.516567 18651 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:57:35.518527 18651 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:57:35.545492 18651 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266: Generated new TSK 0
I20260620 01:57:35.547492 18651 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:57:35.586002 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661
--time_source=system_unsync with env {}
W20260620 01:57:36.182615 18678 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:36.183604 18678 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:36.184031 18678 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:36.196631 18678 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:36.197240 18678 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:36.211285 18678 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:36.214476 18678 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:36.217077 18678 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:36.233155 18683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:36.233142 18684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:36.234424 18686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:36.237067 18678 server_base.cc:1061] running on GCE node
I20260620 01:57:36.238090 18678 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:57:36.238309 18678 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:57:36.238394 18678 hybrid_clock.cc:648] HybridClock initialized: now 1781920656238372 us; error 0 us; skew 500 ppm
I20260620 01:57:36.238758 18678 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:36.243250 18678 webserver.cc:533] Webserver started at http://127.20.1.129:33215/ using document root <none> and password file <none>
I20260620 01:57:36.245930 18678 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:36.246131 18678 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:36.246835 18678 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:36.253064 18678 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "f35e032981c240cab8902ed2c44f025c"
format_stamp: "Formatted at 2026-06-20 01:57:36 on dist-test-slave-wg14"
I20260620 01:57:36.254230 18678 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "f35e032981c240cab8902ed2c44f025c"
format_stamp: "Formatted at 2026-06-20 01:57:36 on dist-test-slave-wg14"
I20260620 01:57:36.261862 18678 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260620 01:57:36.270076 18691 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:36.272956 18678 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.003s sys 0.003s
I20260620 01:57:36.273254 18678 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "f35e032981c240cab8902ed2c44f025c"
format_stamp: "Formatted at 2026-06-20 01:57:36 on dist-test-slave-wg14"
I20260620 01:57:36.273584 18678 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:36.321383 18678 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:36.322814 18678 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:36.323362 18678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:36.326385 18678 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:36.332659 18678 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:36.332824 18678 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:36.332959 18678 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:36.333052 18678 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:36.383003 18678 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:45105
I20260620 01:57:36.383045 18803 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:45105 every 8 connection(s)
I20260620 01:57:36.385052 18678 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:57:36.387387 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 18678
I20260620 01:57:36.388626 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:57:36.434958 18805 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44241
I20260620 01:57:36.435029 18806 heartbeater.cc:344] Connected to a master server at 127.20.1.189:35113
I20260620 01:57:36.434968 18804 heartbeater.cc:344] Connected to a master server at 127.20.1.188:40661
I20260620 01:57:36.435635 18806 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:36.435631 18804 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:36.435752 18805 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:36.438087 18806 heartbeater.cc:507] Master 127.20.1.189:35113 requested a full tablet report, sending...
I20260620 01:57:36.438297 18805 heartbeater.cc:507] Master 127.20.1.190:44241 requested a full tablet report, sending...
I20260620 01:57:36.438318 18804 heartbeater.cc:507] Master 127.20.1.188:40661 requested a full tablet report, sending...
I20260620 01:57:36.441684 18438 ts_manager.cc:194] Registered new tserver with Master: f35e032981c240cab8902ed2c44f025c (127.20.1.129:45105)
I20260620 01:57:36.441758 18581 ts_manager.cc:194] Registered new tserver with Master: f35e032981c240cab8902ed2c44f025c (127.20.1.129:45105)
I20260620 01:57:36.441761 18510 ts_manager.cc:194] Registered new tserver with Master: f35e032981c240cab8902ed2c44f025c (127.20.1.129:45105)
I20260620 01:57:36.444885 18581 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:39901
I20260620 01:57:36.451200 18665 catalog_manager.cc:1294] Loaded cluster ID: d2f491a621eb41edb9eb506862560f94
I20260620 01:57:36.451357 18665 catalog_manager.cc:1587] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5: loading cluster ID for follower catalog manager: success
I20260620 01:57:36.454205 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:57:36.455864 18665 catalog_manager.cc:1609] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5: acquiring CA information for follower catalog manager: success
I20260620 01:57:36.457168 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:41645
--webserver_interface=127.20.1.187
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661,127.20.1.187:41645 with env {}
I20260620 01:57:36.459180 18665 catalog_manager.cc:1637] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:57:36.461205 18669 catalog_manager.cc:1294] Loaded cluster ID: d2f491a621eb41edb9eb506862560f94
I20260620 01:57:36.461381 18669 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499: loading cluster ID for follower catalog manager: success
I20260620 01:57:36.467828 18669 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499: acquiring CA information for follower catalog manager: success
I20260620 01:57:36.471014 18669 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 01:57:37.007506 18815 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:37.008394 18815 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:37.008558 18815 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:37.020107 18815 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:37.020254 18815 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:37.020352 18815 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:37.020430 18815 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:37.036777 18815 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661,127.20.1.187:41645
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:41645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.20.1.187
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:37.043715 18815 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:37.046288 18815 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:37.063205 18820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:37.063849 18821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:37.065220 18823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:37.066812 18815 server_base.cc:1061] running on GCE node
I20260620 01:57:37.068512 18815 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:57:37.068802 18815 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:57:37.069135 18815 hybrid_clock.cc:648] HybridClock initialized: now 1781920657069076 us; error 0 us; skew 500 ppm
I20260620 01:57:37.069931 18815 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:37.074934 18815 webserver.cc:533] Webserver started at http://127.20.1.187:39757/ using document root <none> and password file <none>
I20260620 01:57:37.076161 18815 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:37.076331 18815 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:37.076784 18815 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:37.080682 18815 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/data/instance:
uuid: "e063e2525e0743b080ef78ab3e135cdf"
format_stamp: "Formatted at 2026-06-20 01:57:37 on dist-test-slave-wg14"
I20260620 01:57:37.083009 18815 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/wal/instance:
uuid: "e063e2525e0743b080ef78ab3e135cdf"
format_stamp: "Formatted at 2026-06-20 01:57:37 on dist-test-slave-wg14"
I20260620 01:57:37.093896 18815 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20260620 01:57:37.098907 18828 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:37.103292 18815 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260620 01:57:37.103586 18815 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "e063e2525e0743b080ef78ab3e135cdf"
format_stamp: "Formatted at 2026-06-20 01:57:37 on dist-test-slave-wg14"
I20260620 01:57:37.103900 18815 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:37.133816 18815 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:37.135244 18815 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:37.135587 18815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:37.176433 18815 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:41645
I20260620 01:57:37.176422 18879 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:41645 every 8 connection(s)
I20260620 01:57:37.178490 18815 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 01:57:37.182466 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 18815
I20260620 01:57:37.186873 18880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:57:37.206241 18880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:37.232430 18880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:37.243602 18880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:37.260051 18880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:37.279323 18880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: Bootstrap starting.
I20260620 01:57:37.286198 18880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: Neither blocks nor log segments found. Creating new log.
I20260620 01:57:37.287839 18880 log.cc:826] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:37.292174 18880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: No bootstrap required, opened a new log
I20260620 01:57:37.304878 18880 raft_consensus.cc:359] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } }
I20260620 01:57:37.305439 18880 raft_consensus.cc:385] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:57:37.305547 18880 raft_consensus.cc:740] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063e2525e0743b080ef78ab3e135cdf, State: Initialized, Role: FOLLOWER
I20260620 01:57:37.306632 18880 consensus_queue.cc:260] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } }
I20260620 01:57:37.308173 18889 sys_catalog.cc:455] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } } }
I20260620 01:57:37.308578 18889 sys_catalog.cc:458] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:37.309646 18880 sys_catalog.cc:565] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:37.330608 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 18678
W20260620 01:57:37.333170 18900 catalog_manager.cc:1593] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:57:37.333343 18900 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:57:37.352994 18799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:37.364971 18815 master_runner.cc:429] Detected that this master e063e2525e0743b080ef78ab3e135cdf is joining an existing cluster
I20260620 01:57:37.365105 18815 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260620 01:57:37.366271 18815 raft_consensus.cc:2243] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 01:57:37.367007 18815 raft_consensus.cc:2272] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 01:57:37.367302 18815 tablet_replica.cc:333] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: stopping tablet replica
I20260620 01:57:37.369361 18815 master_runner.cc:434] Initiating AddMaster RPC to add 127.20.1.187:41645
I20260620 01:57:37.471911 18581 catalog_manager.cc:7201] Initiating ChangeConfig request to add master 127.20.1.187:41645: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf"
member_type: NON_VOTER
last_known_addr {
host: "127.20.1.187"
port: 41645
}
attrs {
promote: true
}
}
dest_uuid: "3b2d31e78d1c41fabe77181adc16a266"
cas_config_opid_index: -1
I20260620 01:57:37.475381 18581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: true } }
I20260620 01:57:37.481326 18448 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [term 1 FOLLOWER]: Refusing update from remote peer 3b2d31e78d1c41fabe77181adc16a266: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 01:57:37.484086 18624 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260620 01:57:37.486541 18567 consensus_peers.cc:597] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 -> Peer e063e2525e0743b080ef78ab3e135cdf (127.20.1.187:41645): Couldn't send request to peer e063e2525e0743b080ef78ab3e135cdf. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260620 01:57:37.503535 18624 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e063e2525e0743b080ef78ab3e135cdf (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: true } } }
I20260620 01:57:37.506731 18449 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e063e2525e0743b080ef78ab3e135cdf (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: true } } }
I20260620 01:57:37.510962 18624 catalog_manager.cc:7205] Successfully completed master ChangeConfig request to add master 127.20.1.187:41645
I20260620 01:57:37.514457 18815 master.cc:562] Master@127.20.1.187:41645 shutting down...
I20260620 01:57:37.517999 18916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: true } } }
I20260620 01:57:37.518455 18916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:37.539976 18519 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [term 1 FOLLOWER]: Refusing update from remote peer 3b2d31e78d1c41fabe77181adc16a266: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 01:57:37.555925 18804 heartbeater.cc:499] Master 127.20.1.188:40661 was elected leader, sending a full tablet report...
I20260620 01:57:37.557011 18918 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:57:37.560672 18815 master.cc:584] Master@127.20.1.187:41645 shutdown complete.
I20260620 01:57:37.568478 18815 master_runner.cc:306] Clearing existing system tablet
I20260620 01:57:37.569401 18624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: true } } }
I20260620 01:57:37.570291 18624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:37.570291 18914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: true } } }
I20260620 01:57:37.571381 18914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:37.571725 18815 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:37.571956 18815 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:37.575445 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 18678
I20260620 01:57:37.583664 18815 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 01:57:37.586362 18923 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:37.587287 18815 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:37.587642 18815 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 01:57:37.587771 18815 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "e063e2525e0743b080ef78ab3e135cdf"
format_stamp: "Formatted at 2026-06-20 01:57:37 on dist-test-slave-wg14"
I20260620 01:57:37.592942 18815 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 01:57:37.594412 18519 raft_consensus.cc:2955] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e063e2525e0743b080ef78ab3e135cdf (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: true } } }
I20260620 01:57:37.608239 18815 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 01:57:37.608606 18815 log.cc:1199] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: Deleting WAL directory at /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260620 01:57:37.609297 18815 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: Deleting consensus metadata
I20260620 01:57:37.609606 18815 master_runner.cc:316] Copying system tablet from 127.20.1.188:40661
I20260620 01:57:37.613432 18924 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: true } } }
I20260620 01:57:37.614111 18924 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:37.615692 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:45105
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661
--webserver_port=33215
--webserver_interface=127.20.1.129
--time_source=system_unsync
--tserver_master_addrs=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661,127.20.1.187:41645 with env {}
I20260620 01:57:37.623803 18815 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: tablet copy: Beginning tablet copy session from remote peer at address 127.20.1.188:40661
I20260620 01:57:37.633913 18601 tablet_copy_service.cc:140] P 3b2d31e78d1c41fabe77181adc16a266: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e063e2525e0743b080ef78ab3e135cdf ({username='slave'} at 127.0.0.1:44806)
I20260620 01:57:37.635437 18601 tablet_copy_service.cc:161] P 3b2d31e78d1c41fabe77181adc16a266: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e063e2525e0743b080ef78ab3e135cdf at {username='slave'} at 127.0.0.1:44806: session id = e063e2525e0743b080ef78ab3e135cdf-00000000000000000000000000000000
I20260620 01:57:37.646481 18601 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 01:57:37.651594 18815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 01:57:37.667712 18815 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: tablet copy: Starting download of 0 data blocks...
I20260620 01:57:37.668247 18815 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: tablet copy: Starting download of 1 WAL segments...
I20260620 01:57:37.683404 18815 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 01:57:37.689623 18601 tablet_copy_service.cc:342] P 3b2d31e78d1c41fabe77181adc16a266: Request end of tablet copy session e063e2525e0743b080ef78ab3e135cdf-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:44806
I20260620 01:57:37.690387 18601 tablet_copy_service.cc:434] P 3b2d31e78d1c41fabe77181adc16a266: ending tablet copy session e063e2525e0743b080ef78ab3e135cdf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e063e2525e0743b080ef78ab3e135cdf
I20260620 01:57:37.724885 18815 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:37.744210 18932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:37.747684 18933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:37.755722 18935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:37.758730 18815 server_base.cc:1061] running on GCE node
I20260620 01:57:37.759558 18815 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:57:37.759682 18815 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:57:37.759760 18815 hybrid_clock.cc:648] HybridClock initialized: now 1781920657759731 us; error 0 us; skew 500 ppm
I20260620 01:57:37.759981 18815 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:37.762548 18815 webserver.cc:533] Webserver started at http://127.20.1.187:45681/ using document root <none> and password file <none>
I20260620 01:57:37.763012 18815 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:37.763108 18815 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:37.768517 18815 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 01:57:37.779007 18940 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:37.781919 18815 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.007s sys 0.001s
I20260620 01:57:37.782238 18815 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "e063e2525e0743b080ef78ab3e135cdf"
format_stamp: "Formatted at 2026-06-20 01:57:37 on dist-test-slave-wg14"
I20260620 01:57:37.783818 18815 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:37.818432 18815 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:37.819453 18815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:37.874060 18991 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:41645 every 8 connection(s)
I20260620 01:57:37.873991 18815 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:41645
I20260620 01:57:37.875610 18815 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 01:57:37.886526 18992 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:57:37.890693 18992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: Bootstrap starting.
I20260620 01:57:37.927838 18992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:57:37.928555 18992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: Bootstrap complete.
I20260620 01:57:37.942917 18992 raft_consensus.cc:359] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: true } }
I20260620 01:57:37.943778 18992 raft_consensus.cc:740] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e063e2525e0743b080ef78ab3e135cdf, State: Initialized, Role: LEARNER
I20260620 01:57:37.944321 18992 consensus_queue.cc:260] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: true } }
I20260620 01:57:37.955446 18995 sys_catalog.cc:455] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: true } } }
I20260620 01:57:37.955976 18995 sys_catalog.cc:458] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [sys.catalog]: This master's current role is: LEARNER
I20260620 01:57:37.960610 18992 sys_catalog.cc:565] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:37.994948 18967 raft_consensus.cc:1217] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260620 01:57:38.003901 18995 sys_catalog.cc:455] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b2d31e78d1c41fabe77181adc16a266. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: true } } }
I20260620 01:57:38.004400 18995 sys_catalog.cc:458] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [sys.catalog]: This master's current role is: LEARNER
I20260620 01:57:38.030753 19007 catalog_manager.cc:1294] Loaded cluster ID: d2f491a621eb41edb9eb506862560f94
I20260620 01:57:38.030887 19007 catalog_manager.cc:1587] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: loading cluster ID for follower catalog manager: success
I20260620 01:57:38.036890 19007 catalog_manager.cc:1609] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: acquiring CA information for follower catalog manager: success
I20260620 01:57:38.044472 19007 catalog_manager.cc:1637] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:57:38.066470 18917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: true } } }
I20260620 01:57:38.066951 18917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:38.416863 18919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: true } } }
I20260620 01:57:38.417403 18919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:38.419360 18917 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266: attempting to promote NON_VOTER e063e2525e0743b080ef78ab3e135cdf to VOTER
I20260620 01:57:38.420648 18917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } }
I20260620 01:57:38.423880 18449 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [term 1 FOLLOWER]: Refusing update from remote peer 3b2d31e78d1c41fabe77181adc16a266: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 01:57:38.424228 18519 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [term 1 FOLLOWER]: Refusing update from remote peer 3b2d31e78d1c41fabe77181adc16a266: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 01:57:38.425173 19008 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:57:38.425777 18917 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:57:38.426182 18967 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [term 1 LEARNER]: Refusing update from remote peer 3b2d31e78d1c41fabe77181adc16a266: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 01:57:38.428086 18917 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 01:57:38.432685 19008 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e063e2525e0743b080ef78ab3e135cdf (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } } }
I20260620 01:57:38.434274 18449 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e063e2525e0743b080ef78ab3e135cdf (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } } }
I20260620 01:57:38.440434 18624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } } }
I20260620 01:57:38.441217 18624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:38.442862 18624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } } }
I20260620 01:57:38.443172 18624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:38.444440 18519 raft_consensus.cc:2955] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e063e2525e0743b080ef78ab3e135cdf (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } } }
I20260620 01:57:38.450021 18967 raft_consensus.cc:2955] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e063e2525e0743b080ef78ab3e135cdf (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } } }
I20260620 01:57:38.455755 18995 sys_catalog.cc:455] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } } }
I20260620 01:57:38.456892 19011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } } }
I20260620 01:57:38.457175 19011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:38.457579 18995 sys_catalog.cc:458] T 00000000000000000000000000000000 P e063e2525e0743b080ef78ab3e135cdf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:38.459730 18919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } } }
I20260620 01:57:38.459769 19010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } } }
I20260620 01:57:38.460278 18919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:38.460279 19010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:38.462316 19012 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } } }
I20260620 01:57:38.462734 19012 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffc8f7d4da14530893ed18893e13ed5 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:57:38.482753 18931 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:38.483268 18931 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:38.483402 18931 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:38.497282 18931 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:38.497494 18931 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:38.519590 18931 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:45105
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=33215
--tserver_master_addrs=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661,127.20.1.187:41645
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:38.523805 18931 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:38.526103 18931 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:38.544523 19017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:38.544893 19018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:38.547477 19020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:38.547885 18931 server_base.cc:1061] running on GCE node
I20260620 01:57:38.548695 18931 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:57:38.548907 18931 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:57:38.549000 18931 hybrid_clock.cc:648] HybridClock initialized: now 1781920658548976 us; error 0 us; skew 500 ppm
I20260620 01:57:38.549355 18931 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:38.553020 18931 webserver.cc:533] Webserver started at http://127.20.1.129:33215/ using document root <none> and password file <none>
I20260620 01:57:38.554102 18931 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:38.554277 18931 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:38.562422 18931 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.004s
I20260620 01:57:38.568378 19025 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:38.570614 18931 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260620 01:57:38.570850 18931 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "f35e032981c240cab8902ed2c44f025c"
format_stamp: "Formatted at 2026-06-20 01:57:36 on dist-test-slave-wg14"
I20260620 01:57:38.571882 18931 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:38.629707 18931 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:38.630939 18931 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:38.631357 18931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:38.632676 18931 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:38.635238 18931 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:38.635357 18931 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 01:57:38.635475 18931 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:38.635522 18931 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:38.709566 18931 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:45105
I20260620 01:57:38.709573 19137 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:45105 every 8 connection(s)
I20260620 01:57:38.712193 18931 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:57:38.716776 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 18931
I20260620 01:57:38.768512 19138 heartbeater.cc:344] Connected to a master server at 127.20.1.188:40661
I20260620 01:57:38.769220 19138 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:38.771062 19138 heartbeater.cc:507] Master 127.20.1.188:40661 requested a full tablet report, sending...
I20260620 01:57:38.772876 19141 heartbeater.cc:344] Connected to a master server at 127.20.1.189:35113
I20260620 01:57:38.773187 19140 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44241
I20260620 01:57:38.773437 19141 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:38.773437 19140 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:38.774223 18581 ts_manager.cc:194] Re-registered known tserver with Master: f35e032981c240cab8902ed2c44f025c (127.20.1.129:45105)
I20260620 01:57:38.774747 19140 heartbeater.cc:507] Master 127.20.1.190:44241 requested a full tablet report, sending...
I20260620 01:57:38.774782 19141 heartbeater.cc:507] Master 127.20.1.189:35113 requested a full tablet report, sending...
I20260620 01:57:38.775483 18581 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:51463
I20260620 01:57:38.777519 18438 ts_manager.cc:194] Re-registered known tserver with Master: f35e032981c240cab8902ed2c44f025c (127.20.1.129:45105)
I20260620 01:57:38.778136 19139 heartbeater.cc:344] Connected to a master server at 127.20.1.187:41645
I20260620 01:57:38.778355 19139 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:38.779210 19139 heartbeater.cc:507] Master 127.20.1.187:41645 requested a full tablet report, sending...
I20260620 01:57:38.779729 18510 ts_manager.cc:194] Re-registered known tserver with Master: f35e032981c240cab8902ed2c44f025c (127.20.1.129:45105)
I20260620 01:57:38.781368 18957 ts_manager.cc:194] Registered new tserver with Master: f35e032981c240cab8902ed2c44f025c (127.20.1.129:45105)
I20260620 01:57:38.796615 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:57:38.816459 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:57:38.816802 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 18409
I20260620 01:57:38.818274 18469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:38.954689 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 18409
I20260620 01:57:38.982453 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44241
--webserver_interface=127.20.1.190
--webserver_port=40113
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661
--master_addresses=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661,127.20.1.187:41645 with env {}
W20260620 01:57:39.317376 18568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.190:44241: connect: Connection refused (error 111)
W20260620 01:57:39.322367 18568 consensus_peers.cc:597] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 -> Peer 10dc7a82a39746cbbe804607533b5499 (127.20.1.190:44241): Couldn't send request to peer 10dc7a82a39746cbbe804607533b5499. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:44241: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260620 01:57:39.488415 19154 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:39.489315 19154 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:39.489418 19154 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:39.499876 19154 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:39.500026 19154 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:39.500095 19154 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:39.500137 19154 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:39.516145 19154 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661,127.20.1.187:41645
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=40113
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:39.518195 19154 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:39.520571 19154 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:39.535651 19162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:39.535751 19164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:39.535583 19161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:39.538661 19154 server_base.cc:1061] running on GCE node
I20260620 01:57:39.539955 19154 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:57:39.540308 19154 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:57:39.540524 19154 hybrid_clock.cc:648] HybridClock initialized: now 1781920659540497 us; error 0 us; skew 500 ppm
I20260620 01:57:39.540980 19154 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:39.545517 19154 webserver.cc:533] Webserver started at http://127.20.1.190:40113/ using document root <none> and password file <none>
I20260620 01:57:39.546625 19154 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:39.547019 19154 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:39.557741 19154 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.007s sys 0.000s
I20260620 01:57:39.563673 19169 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:39.566778 19154 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260620 01:57:39.567073 19154 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "10dc7a82a39746cbbe804607533b5499"
format_stamp: "Formatted at 2026-06-20 01:57:33 on dist-test-slave-wg14"
I20260620 01:57:39.568123 19154 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:39.604696 19154 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:39.605980 19154 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:39.607157 19154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:39.637315 19154 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44241
I20260620 01:57:39.637351 19220 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44241 every 8 connection(s)
I20260620 01:57:39.639544 19154 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:57:39.644065 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 19154
I20260620 01:57:39.644464 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 18931
I20260620 01:57:39.650487 19221 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:57:39.664553 19133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:39.664604 19221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499: Bootstrap starting.
I20260620 01:57:39.703055 19221 log.cc:826] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:39.732039 19221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:57:39.733716 19221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499: Bootstrap complete.
I20260620 01:57:39.748786 19221 raft_consensus.cc:359] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } }
I20260620 01:57:39.749881 19221 raft_consensus.cc:740] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10dc7a82a39746cbbe804607533b5499, State: Initialized, Role: FOLLOWER
I20260620 01:57:39.751016 19221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } }
I20260620 01:57:39.752275 19227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } } }
I20260620 01:57:39.752717 19227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:39.753811 19221 sys_catalog.cc:565] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:39.771862 19238 catalog_manager.cc:1294] Loaded cluster ID: d2f491a621eb41edb9eb506862560f94
I20260620 01:57:39.772137 19238 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499: loading cluster ID for follower catalog manager: success
I20260620 01:57:39.781517 19238 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499: acquiring CA information for follower catalog manager: success
I20260620 01:57:39.786422 19238 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:57:39.813900 19138 heartbeater.cc:499] Master 127.20.1.188:40661 was elected leader, sending a full tablet report...
I20260620 01:57:39.816648 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 18931
I20260620 01:57:39.820479 18919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } } }
I20260620 01:57:39.820809 18919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: This master's current role is: LEADER
W20260620 01:57:39.853263 18568 connection.cc:573] server connection from 127.20.1.129:51463 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:57:39.855096 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:45105
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661
--webserver_port=33215
--webserver_interface=127.20.1.129
--time_source=system_unsync
--tserver_master_addrs=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661,127.20.1.187:41645 with env {}
I20260620 01:57:39.858340 19227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b2d31e78d1c41fabe77181adc16a266. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } } }
I20260620 01:57:39.859853 19227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10dc7a82a39746cbbe804607533b5499 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:57:39.864353 19241 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:57:39.819878 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:42065 (local address 127.20.1.190:44241)
0620 01:57:39.827245 (+ 7367us) server_negotiation.cc:207] Beginning negotiation
0620 01:57:39.827268 (+ 23us) server_negotiation.cc:400] Waiting for connection header
0620 01:57:39.827310 (+ 42us) server_negotiation.cc:408] Connection header received
0620 01:57:39.855747 (+ 28437us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:42065: BlockingRecv error: recv got EOF from 127.20.1.129:42065 (error 108)
Metrics: {"server-negotiator.queue_time_us":7101}
I20260620 01:57:40.359897 18919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b2d31e78d1c41fabe77181adc16a266" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10dc7a82a39746cbbe804607533b5499" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44241 } } peers { permanent_uuid: "fffc8f7d4da14530893ed18893e13ed5" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 35113 } } peers { permanent_uuid: "3b2d31e78d1c41fabe77181adc16a266" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40661 } } peers { permanent_uuid: "e063e2525e0743b080ef78ab3e135cdf" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41645 } attrs { promote: false } } }
I20260620 01:57:40.360275 18919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 [sys.catalog]: This master's current role is: LEADER
W20260620 01:57:40.425292 19248 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:40.425836 19248 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:40.425993 19248 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:40.441792 19248 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:40.442054 19248 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:40.458539 19248 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:45105
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=33215
--tserver_master_addrs=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661,127.20.1.187:41645
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:40.461015 19248 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:40.464440 19248 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:40.480618 19260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:40.481034 19259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:40.484360 19262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:40.486109 19248 server_base.cc:1061] running on GCE node
I20260620 01:57:40.487067 19248 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:57:40.487378 19248 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:57:40.487506 19248 hybrid_clock.cc:648] HybridClock initialized: now 1781920660487467 us; error 0 us; skew 500 ppm
I20260620 01:57:40.487967 19248 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:40.491889 19248 webserver.cc:533] Webserver started at http://127.20.1.129:33215/ using document root <none> and password file <none>
I20260620 01:57:40.493428 19248 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:40.493654 19248 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:40.502252 19248 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 01:57:40.507743 19267 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:40.510111 19248 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260620 01:57:40.510386 19248 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "f35e032981c240cab8902ed2c44f025c"
format_stamp: "Formatted at 2026-06-20 01:57:36 on dist-test-slave-wg14"
I20260620 01:57:40.512296 19248 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:40.562197 19248 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:40.565465 19248 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:40.566637 19248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:40.568562 19248 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:40.571503 19248 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:40.571635 19248 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:40.571760 19248 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:40.571828 19248 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:40.651943 19248 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:45105
I20260620 01:57:40.651980 19379 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:45105 every 8 connection(s)
I20260620 01:57:40.654167 19248 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:57:40.656081 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 19248
I20260620 01:57:40.656524 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 19248
I20260620 01:57:40.703951 19380 heartbeater.cc:344] Connected to a master server at 127.20.1.188:40661
I20260620 01:57:40.704110 19382 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44241
I20260620 01:57:40.704471 19380 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:40.704464 19382 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:40.706221 19380 heartbeater.cc:507] Master 127.20.1.188:40661 requested a full tablet report, sending...
I20260620 01:57:40.706387 19385 heartbeater.cc:344] Connected to a master server at 127.20.1.189:35113
I20260620 01:57:40.706497 19382 heartbeater.cc:507] Master 127.20.1.190:44241 requested a full tablet report, sending...
I20260620 01:57:40.706653 19385 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:40.707765 19385 heartbeater.cc:507] Master 127.20.1.189:35113 requested a full tablet report, sending...
I20260620 01:57:40.709993 19375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:40.710052 19186 ts_manager.cc:194] Registered new tserver with Master: f35e032981c240cab8902ed2c44f025c (127.20.1.129:45105)
I20260620 01:57:40.712016 18510 ts_manager.cc:194] Re-registered known tserver with Master: f35e032981c240cab8902ed2c44f025c (127.20.1.129:45105)
I20260620 01:57:40.713363 18581 ts_manager.cc:194] Re-registered known tserver with Master: f35e032981c240cab8902ed2c44f025c (127.20.1.129:45105)
I20260620 01:57:40.714777 18581 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:46905
I20260620 01:57:40.826483 19381 heartbeater.cc:344] Connected to a master server at 127.20.1.187:41645
I20260620 01:57:40.826746 19381 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:40.826889 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 19248
I20260620 01:57:40.846426 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:45105
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661
--webserver_port=33215
--webserver_interface=127.20.1.129
--time_source=system_unsync
--tserver_master_addrs=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661,127.20.1.187:41645
--fault_before_start=1.0 with env {}
W20260620 01:57:41.406975 19395 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:41.407488 19395 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:41.407634 19395 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:57:41.407718 19395 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:41.418820 19395 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:41.419472 19395 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:41.434494 19395 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:45105
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=33215
--tserver_master_addrs=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661,127.20.1.187:41645
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:41.436820 19395 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:41.439165 19395 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:41.454154 19401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:41.454474 19403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:41.454144 19400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:41.457582 19395 server_base.cc:1061] running on GCE node
I20260620 01:57:41.459487 19395 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:57:41.459966 19395 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:57:41.460148 19395 hybrid_clock.cc:648] HybridClock initialized: now 1781920661460087 us; error 0 us; skew 500 ppm
I20260620 01:57:41.460819 19395 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:41.464685 19395 webserver.cc:533] Webserver started at http://127.20.1.129:33215/ using document root <none> and password file <none>
I20260620 01:57:41.466475 19395 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:41.466733 19395 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:41.475731 19395 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 01:57:41.480638 19408 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:41.483593 19395 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:57:41.484084 19395 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "f35e032981c240cab8902ed2c44f025c"
format_stamp: "Formatted at 2026-06-20 01:57:36 on dist-test-slave-wg14"
I20260620 01:57:41.485584 19395 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:41.517729 19395 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:41.519002 19395 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:41.519395 19395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:41.521726 19395 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:41.525005 19395 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:41.525305 19395 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:41.525506 19395 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:41.525597 19395 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:57:41.526196 19395 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:57:41.526309 19395 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:57:41.526363 19395 fault_injection.cc:59] injecting fault for kudu (pid 19395): FLAGS_fault_before_start (process will exit)
I20260620 01:57:41.551403 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:45105
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661
--webserver_port=33215
--webserver_interface=127.20.1.129
--time_source=system_unsync
--tserver_master_addrs=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661,127.20.1.187:41645 with env {}
W20260620 01:57:42.077528 19416 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:42.077929 19416 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:42.078049 19416 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:42.088577 19416 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:42.088812 19416 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:42.102029 19416 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:45105
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=33215
--tserver_master_addrs=127.20.1.190:44241,127.20.1.189:35113,127.20.1.188:40661,127.20.1.187:41645
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:42.104441 19416 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:42.109052 19416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:42.125815 19421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:42.126091 19422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:42.129763 19416 server_base.cc:1061] running on GCE node
W20260620 01:57:42.130034 19424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:42.131721 19416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:57:42.132400 19416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:57:42.132730 19416 hybrid_clock.cc:648] HybridClock initialized: now 1781920662132695 us; error 0 us; skew 500 ppm
I20260620 01:57:42.133209 19416 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:42.141541 19416 webserver.cc:533] Webserver started at http://127.20.1.129:33215/ using document root <none> and password file <none>
I20260620 01:57:42.142859 19416 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:42.143092 19416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:42.152773 19416 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 01:57:42.157851 19429 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:42.160259 19416 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.004s
I20260620 01:57:42.160511 19416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "f35e032981c240cab8902ed2c44f025c"
format_stamp: "Formatted at 2026-06-20 01:57:36 on dist-test-slave-wg14"
I20260620 01:57:42.161374 19416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:42.202782 19416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:42.205644 19416 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:42.206100 19416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:42.207614 19416 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:42.210391 19416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:42.210521 19416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:42.210671 19416 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:42.210737 19416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:42.269863 19416 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:45105
I20260620 01:57:42.269889 19541 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:45105 every 8 connection(s)
I20260620 01:57:42.273985 19416 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:57:42.277349 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 19416
I20260620 01:57:42.277801 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 19416
I20260620 01:57:42.328869 19543 heartbeater.cc:344] Connected to a master server at 127.20.1.187:41645
I20260620 01:57:42.329347 19543 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:42.330042 19537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:42.456153 19543 heartbeater.cc:507] Master 127.20.1.187:41645 requested a full tablet report, sending...
I20260620 01:57:42.458106 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 19416
I20260620 01:57:42.461638 18957 ts_manager.cc:194] Re-registered known tserver with Master: f35e032981c240cab8902ed2c44f025c (127.20.1.129:45105)
W20260620 01:57:42.479020 19172 connection.cc:573] server connection from 127.20.1.129:48429 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 01:57:42.479424 18943 connection.cc:573] server connection from 127.20.1.129:49467 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:57:42.479882 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 19154
W20260620 01:57:42.481102 19555 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:57:42.325356 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:40955 (local address 127.20.1.188:40661)
0620 01:57:42.331220 (+ 5864us) server_negotiation.cc:207] Beginning negotiation
0620 01:57:42.331227 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 01:57:42.331247 (+ 20us) server_negotiation.cc:408] Connection header received
0620 01:57:42.331318 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:57:42.331322 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:57:42.331473 (+ 151us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:57:42.331972 (+ 499us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:57:42.456481 (+124509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:57:42.458418 (+ 1937us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:57:42.479769 (+ 21351us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:40955: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":5679,"thread_start_us":6206,"threads_started":1}
W20260620 01:57:42.481911 19551 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:57:42.286636 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:50659 (local address 127.20.1.189:35113)
0620 01:57:42.322429 (+ 35793us) server_negotiation.cc:207] Beginning negotiation
0620 01:57:42.322437 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:57:42.322461 (+ 24us) server_negotiation.cc:408] Connection header received
0620 01:57:42.322548 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:57:42.322553 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:57:42.322669 (+ 116us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:57:42.322812 (+ 143us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:57:42.325302 (+ 2490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:57:42.329485 (+ 4183us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:57:42.479462 (+149977us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:50659: BlockingRecv error: recv got EOF from 127.20.1.129:50659 (error 108)
Metrics: {"server-negotiator.queue_time_us":35623,"thread_start_us":35522,"threads_started":1}
I20260620 01:57:42.490636 19216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:42.653144 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 19154
I20260620 01:57:42.680600 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 18476
I20260620 01:57:42.682384 18540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:42.817816 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 18476
W20260620 01:57:42.834473 18568 consensus_peers.cc:597] T 00000000000000000000000000000000 P 3b2d31e78d1c41fabe77181adc16a266 -> Peer 10dc7a82a39746cbbe804607533b5499 (127.20.1.190:44241): Couldn't send request to peer 10dc7a82a39746cbbe804607533b5499. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:44241: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 01:57:42.845860 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 18547
I20260620 01:57:42.847801 18611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:43.015211 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 18547
W20260620 01:57:43.046418 18944 connection.cc:573] server connection from 127.0.0.1:39602 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:57:43.046916 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 18815
I20260620 01:57:43.048648 18987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:43.197729 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 18815
[ OK ] ExternalMiniClusterTest.TestAddMaster/0 (10850 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/1
2026-06-20T01:57:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:57:43Z Disabled control of system clock
I20260620 01:57:43.294643 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:33935
--webserver_interface=127.20.1.190
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459 with env {}
W20260620 01:57:43.780893 19566 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:43.781385 19566 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:43.781713 19566 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:43.792214 19566 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:43.792352 19566 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:43.792409 19566 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:43.792459 19566 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:43.805305 19566 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:33935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:43.807816 19566 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:43.810457 19566 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:43.823297 19571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:43.823246 19572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:43.823581 19574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:43.825775 19566 server_base.cc:1061] running on GCE node
I20260620 01:57:43.826963 19566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:43.829056 19566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:43.830397 19566 hybrid_clock.cc:648] HybridClock initialized: now 1781920663830310 us; error 87 us; skew 500 ppm
I20260620 01:57:43.830858 19566 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:43.834622 19566 webserver.cc:533] Webserver started at http://127.20.1.190:46671/ using document root <none> and password file <none>
I20260620 01:57:43.836539 19566 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:43.836762 19566 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:43.837285 19566 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:43.841130 19566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "95e50c60221447b79e64a3c0ceac162f"
format_stamp: "Formatted at 2026-06-20 01:57:43 on dist-test-slave-wg14"
I20260620 01:57:43.842357 19566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "95e50c60221447b79e64a3c0ceac162f"
format_stamp: "Formatted at 2026-06-20 01:57:43 on dist-test-slave-wg14"
I20260620 01:57:43.850332 19566 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260620 01:57:43.856526 19580 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:43.859293 19566 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:57:43.859798 19566 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "95e50c60221447b79e64a3c0ceac162f"
format_stamp: "Formatted at 2026-06-20 01:57:43 on dist-test-slave-wg14"
I20260620 01:57:43.860299 19566 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:43.884554 19566 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:43.885870 19566 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:43.886238 19566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:43.914214 19566 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:33935
I20260620 01:57:43.914261 19631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:33935 every 8 connection(s)
I20260620 01:57:43.916791 19566 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:57:43.921733 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 19566
I20260620 01:57:43.921967 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:57:43.923496 19632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:57:43.923496 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:45367
--webserver_interface=127.20.1.189
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459 with env {}
I20260620 01:57:43.933669 19632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:43.968952 19632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:57:43.970661 19583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45367: connect: Connection refused (error 111)
W20260620 01:57:43.973285 19632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:45367: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45367: connect: Connection refused (error 111)
I20260620 01:57:44.022792 19632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } attempt: 1
W20260620 01:57:44.025542 19632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:45367: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45367: connect: Connection refused (error 111)
I20260620 01:57:44.094216 19632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } attempt: 2
W20260620 01:57:44.097877 19632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:45367: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45367: connect: Connection refused (error 111)
I20260620 01:57:44.189359 19632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } attempt: 3
W20260620 01:57:44.192663 19632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:45367: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45367: connect: Connection refused (error 111)
I20260620 01:57:44.336369 19632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } attempt: 4
W20260620 01:57:44.343897 19632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:45367: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45367: connect: Connection refused (error 111)
W20260620 01:57:44.501955 19634 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:44.502513 19634 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:44.502648 19634 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:44.516350 19634 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:44.516474 19634 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:44.516558 19634 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:44.516625 19634 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:44.531114 19634 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:45367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:44.533564 19634 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:44.536242 19634 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:44.549005 19642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:44.548480 19643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:44.551373 19645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:44.554349 19634 server_base.cc:1061] running on GCE node
I20260620 01:57:44.555635 19634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:44.557792 19634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:44.559309 19634 hybrid_clock.cc:648] HybridClock initialized: now 1781920664559224 us; error 65 us; skew 500 ppm
I20260620 01:57:44.559866 19634 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:44.564452 19634 webserver.cc:533] Webserver started at http://127.20.1.189:38665/ using document root <none> and password file <none>
I20260620 01:57:44.565982 19634 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:44.566248 19634 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:44.567394 19634 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:44.573064 19634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "96ef7785a2024d13bf6ab7bb8028039a"
format_stamp: "Formatted at 2026-06-20 01:57:44 on dist-test-slave-wg14"
I20260620 01:57:44.574385 19634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "96ef7785a2024d13bf6ab7bb8028039a"
format_stamp: "Formatted at 2026-06-20 01:57:44 on dist-test-slave-wg14"
I20260620 01:57:44.585806 19634 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.004s sys 0.004s
I20260620 01:57:44.592358 19651 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:44.595448 19634 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:57:44.595808 19634 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "96ef7785a2024d13bf6ab7bb8028039a"
format_stamp: "Formatted at 2026-06-20 01:57:44 on dist-test-slave-wg14"
I20260620 01:57:44.596146 19634 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:44.622670 19634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:44.624161 19634 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:44.624514 19634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:44.643316 19632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } attempt: 5
W20260620 01:57:44.646361 19632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:45367: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45367: connect: Connection refused (error 111)
I20260620 01:57:44.658735 19634 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:45367
I20260620 01:57:44.658921 19702 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:45367 every 8 connection(s)
I20260620 01:57:44.661613 19634 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:57:44.668078 19703 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:57:44.669987 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 19634
I20260620 01:57:44.670297 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:57:44.671772 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:39459
--webserver_interface=127.20.1.188
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459 with env {}
I20260620 01:57:44.679387 19703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:44.708367 19703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:44.725018 19703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:57:44.727228 19654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39459: connect: Connection refused (error 111)
W20260620 01:57:44.729496 19703 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:39459: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39459: connect: Connection refused (error 111)
I20260620 01:57:44.778972 19703 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } attempt: 1
W20260620 01:57:44.782670 19703 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:39459: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39459: connect: Connection refused (error 111)
I20260620 01:57:44.851257 19703 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } attempt: 2
W20260620 01:57:44.854847 19703 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:39459: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39459: connect: Connection refused (error 111)
I20260620 01:57:44.946309 19703 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } attempt: 3
W20260620 01:57:44.949517 19703 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:39459: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39459: connect: Connection refused (error 111)
I20260620 01:57:45.092926 19703 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } attempt: 4
W20260620 01:57:45.096997 19703 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:39459: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39459: connect: Connection refused (error 111)
W20260620 01:57:45.179360 19707 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:45.179787 19707 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:45.179868 19707 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:45.190388 19707 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:45.190521 19707 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:45.190578 19707 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:45.190620 19707 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:45.193760 19632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } attempt: 6
I20260620 01:57:45.206753 19707 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:39459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:45.207616 19632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:45.210594 19707 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260620 01:57:45.211529 19632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:39459: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39459: connect: Connection refused (error 111)
I20260620 01:57:45.213863 19707 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:45.225852 19716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:45.226230 19717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:45.227363 19719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:45.228538 19707 server_base.cc:1061] running on GCE node
I20260620 01:57:45.230329 19707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:45.233597 19707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:45.235155 19707 hybrid_clock.cc:648] HybridClock initialized: now 1781920665235049 us; error 71 us; skew 500 ppm
I20260620 01:57:45.235731 19707 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:45.239457 19707 webserver.cc:533] Webserver started at http://127.20.1.188:45195/ using document root <none> and password file <none>
I20260620 01:57:45.240520 19707 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:45.240687 19707 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:45.241123 19707 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:45.244299 19707 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "0c207000b0c44c5ca7d8575102e1d63a"
format_stamp: "Formatted at 2026-06-20 01:57:45 on dist-test-slave-wg14"
I20260620 01:57:45.245812 19707 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "0c207000b0c44c5ca7d8575102e1d63a"
format_stamp: "Formatted at 2026-06-20 01:57:45 on dist-test-slave-wg14"
I20260620 01:57:45.255729 19707 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.010s sys 0.000s
I20260620 01:57:45.260798 19725 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:45.262800 19707 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 01:57:45.263229 19707 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "0c207000b0c44c5ca7d8575102e1d63a"
format_stamp: "Formatted at 2026-06-20 01:57:45 on dist-test-slave-wg14"
I20260620 01:57:45.263500 19707 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:45.263850 19632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } attempt: 1
W20260620 01:57:45.266398 19632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:39459: Network error: Client connection negotiation failed: client connection to 127.20.1.188:39459: connect: Connection refused (error 111)
I20260620 01:57:45.286302 19707 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:45.287647 19707 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:45.287986 19707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:45.323333 19707 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:39459
I20260620 01:57:45.323896 19776 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:39459 every 8 connection(s)
I20260620 01:57:45.325577 19707 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:57:45.332291 19777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:57:45.335537 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 19707
I20260620 01:57:45.335742 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:57:45.341305 19632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } attempt: 2
I20260620 01:57:45.353497 19777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:45.394490 19777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:45.396431 19703 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } attempt: 5
I20260620 01:57:45.409842 19632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f: Bootstrap starting.
I20260620 01:57:45.410923 19777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:45.414783 19703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a: Bootstrap starting.
I20260620 01:57:45.416901 19632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f: Neither blocks nor log segments found. Creating new log.
I20260620 01:57:45.419293 19632 log.cc:826] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:45.422672 19703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a: Neither blocks nor log segments found. Creating new log.
I20260620 01:57:45.426338 19703 log.cc:826] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:45.429479 19777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a: Bootstrap starting.
I20260620 01:57:45.433434 19703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a: No bootstrap required, opened a new log
I20260620 01:57:45.433425 19632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f: No bootstrap required, opened a new log
I20260620 01:57:45.433892 19777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a: Neither blocks nor log segments found. Creating new log.
I20260620 01:57:45.436125 19777 log.cc:826] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:45.442543 19632 raft_consensus.cc:359] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } }
I20260620 01:57:45.443081 19632 raft_consensus.cc:385] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:57:45.443308 19632 raft_consensus.cc:740] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e50c60221447b79e64a3c0ceac162f, State: Initialized, Role: FOLLOWER
I20260620 01:57:45.444392 19632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } }
I20260620 01:57:45.444779 19777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a: No bootstrap required, opened a new log
I20260620 01:57:45.444972 19703 raft_consensus.cc:359] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } }
I20260620 01:57:45.445540 19703 raft_consensus.cc:385] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:57:45.445737 19703 raft_consensus.cc:740] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96ef7785a2024d13bf6ab7bb8028039a, State: Initialized, Role: FOLLOWER
I20260620 01:57:45.446733 19788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } }
I20260620 01:57:45.446866 19703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } }
I20260620 01:57:45.447480 19788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:45.448781 19632 sys_catalog.cc:565] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:45.448593 19789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } }
I20260620 01:57:45.449259 19789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:45.450407 19703 sys_catalog.cc:565] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:45.456267 19777 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } }
I20260620 01:57:45.456904 19777 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:57:45.457469 19777 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c207000b0c44c5ca7d8575102e1d63a, State: Initialized, Role: FOLLOWER
I20260620 01:57:45.459653 19777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } }
I20260620 01:57:45.468108 19795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } }
I20260620 01:57:45.468663 19795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:45.469581 19777 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:45.478806 19788 raft_consensus.cc:493] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:57:45.479238 19788 raft_consensus.cc:515] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } }
I20260620 01:57:45.481638 19788 leader_election.cc:290] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96ef7785a2024d13bf6ab7bb8028039a (127.20.1.189:45367), 0c207000b0c44c5ca7d8575102e1d63a (127.20.1.188:39459)
I20260620 01:57:45.483781 19752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95e50c60221447b79e64a3c0ceac162f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c207000b0c44c5ca7d8575102e1d63a" is_pre_election: true
I20260620 01:57:45.483863 19678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95e50c60221447b79e64a3c0ceac162f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96ef7785a2024d13bf6ab7bb8028039a" is_pre_election: true
I20260620 01:57:45.484736 19752 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95e50c60221447b79e64a3c0ceac162f in term 0.
I20260620 01:57:45.484789 19678 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95e50c60221447b79e64a3c0ceac162f in term 0.
I20260620 01:57:45.486294 19583 leader_election.cc:304] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95e50c60221447b79e64a3c0ceac162f, 96ef7785a2024d13bf6ab7bb8028039a; no voters:
I20260620 01:57:45.487178 19788 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:57:45.487584 19788 raft_consensus.cc:493] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:57:45.487766 19788 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:57:45.492391 19788 raft_consensus.cc:515] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } }
I20260620 01:57:45.504323 19788 leader_election.cc:290] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [CANDIDATE]: Term 1 election: Requested vote from peers 96ef7785a2024d13bf6ab7bb8028039a (127.20.1.189:45367), 0c207000b0c44c5ca7d8575102e1d63a (127.20.1.188:39459)
I20260620 01:57:45.505251 19678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95e50c60221447b79e64a3c0ceac162f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96ef7785a2024d13bf6ab7bb8028039a"
I20260620 01:57:45.505748 19678 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:57:45.506070 19752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95e50c60221447b79e64a3c0ceac162f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c207000b0c44c5ca7d8575102e1d63a"
I20260620 01:57:45.506322 19752 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:57:45.510883 19752 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e50c60221447b79e64a3c0ceac162f in term 1.
I20260620 01:57:45.510860 19678 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e50c60221447b79e64a3c0ceac162f in term 1.
I20260620 01:57:45.504464 19813 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 01:57:45.512171 19583 leader_election.cc:304] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95e50c60221447b79e64a3c0ceac162f, 96ef7785a2024d13bf6ab7bb8028039a; no voters:
I20260620 01:57:45.515986 19788 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:57:45.527315 19824 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 01:57:45.530309 19788 raft_consensus.cc:697] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 1 LEADER]: Becoming Leader. State: Replica: 95e50c60221447b79e64a3c0ceac162f, State: Running, Role: LEADER
I20260620 01:57:45.531232 19788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } }
W20260620 01:57:45.534487 19819 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:57:45.534786 19819 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:57:45.538908 19820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95e50c60221447b79e64a3c0ceac162f. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } }
I20260620 01:57:45.539413 19820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:45.540892 19827 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
W20260620 01:57:45.542945 19731 tablet.cc:2406] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:57:45.550720 19827 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:57:45.584935 19752 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 1 FOLLOWER]: Refusing update from remote peer 95e50c60221447b79e64a3c0ceac162f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:57:45.584941 19678 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 1 FOLLOWER]: Refusing update from remote peer 95e50c60221447b79e64a3c0ceac162f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:57:45.586637 19788 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [LEADER]: Connected to new peer: Peer: permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:57:45.587899 19788 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:57:45.600548 19795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95e50c60221447b79e64a3c0ceac162f. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } }
I20260620 01:57:45.600889 19795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:45.604791 19829 mvcc.cc:204] Tried to move back new op lower bound from 7298747046203650048 to 7298747046022475776. Current Snapshot: MvccSnapshot[applied={T|T < 7298747046203650048}]
I20260620 01:57:45.605728 19830 mvcc.cc:204] Tried to move back new op lower bound from 7298747046203650048 to 7298747046022475776. Current Snapshot: MvccSnapshot[applied={T|T < 7298747046203650048}]
I20260620 01:57:45.606680 19795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } }
I20260620 01:57:45.607041 19795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:45.626157 19827 catalog_manager.cc:1382] Generated new cluster ID: 8fe3ea9f935e40efbc81cc6dfa54c5a2
I20260620 01:57:45.626540 19827 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
W20260620 01:57:45.628443 19657 tablet.cc:2406] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:57:45.628482 19789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95e50c60221447b79e64a3c0ceac162f. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } }
I20260620 01:57:45.630270 19789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:45.639803 19820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } }
I20260620 01:57:45.640262 19820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:45.640690 19820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } }
I20260620 01:57:45.640908 19820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:45.652319 19789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } }
I20260620 01:57:45.652628 19789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:45.684823 19827 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:57:45.687395 19827 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:57:45.720414 19827 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f: Generated new TSK 0
I20260620 01:57:45.722611 19827 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:57:45.744877 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459
--builtin_ntp_servers=127.20.1.148:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260620 01:57:46.530133 19838 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:46.530591 19838 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:46.530731 19838 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260620 01:57:46.534952 19824 catalog_manager.cc:1294] Loaded cluster ID: 8fe3ea9f935e40efbc81cc6dfa54c5a2
I20260620 01:57:46.535215 19824 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a: loading cluster ID for follower catalog manager: success
I20260620 01:57:46.538096 19819 catalog_manager.cc:1294] Loaded cluster ID: 8fe3ea9f935e40efbc81cc6dfa54c5a2
I20260620 01:57:46.538410 19819 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a: loading cluster ID for follower catalog manager: success
I20260620 01:57:46.540539 19824 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a: acquiring CA information for follower catalog manager: success
I20260620 01:57:46.544432 19824 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:57:46.545113 19819 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a: acquiring CA information for follower catalog manager: success
W20260620 01:57:46.545864 19838 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:46.546314 19838 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:46.550283 19819 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:57:46.561393 19838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:46.563611 19838 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:46.566402 19838 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:46.583931 19844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:46.586191 19845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:46.590404 19847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:46.591243 19838 server_base.cc:1061] running on GCE node
I20260620 01:57:46.592633 19838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:46.594632 19838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:46.596107 19838 hybrid_clock.cc:648] HybridClock initialized: now 1781920666595994 us; error 114 us; skew 500 ppm
I20260620 01:57:46.597048 19838 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:46.600356 19838 webserver.cc:533] Webserver started at http://127.20.1.129:42387/ using document root <none> and password file <none>
I20260620 01:57:46.601468 19838 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:46.601647 19838 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:46.602170 19838 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:46.606376 19838 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "7fa3c745ff6c4a3ea05064d339d1c4a9"
format_stamp: "Formatted at 2026-06-20 01:57:46 on dist-test-slave-wg14"
I20260620 01:57:46.608009 19838 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "7fa3c745ff6c4a3ea05064d339d1c4a9"
format_stamp: "Formatted at 2026-06-20 01:57:46 on dist-test-slave-wg14"
I20260620 01:57:46.616776 19838 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260620 01:57:46.622094 19853 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:46.624423 19838 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:57:46.624714 19838 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "7fa3c745ff6c4a3ea05064d339d1c4a9"
format_stamp: "Formatted at 2026-06-20 01:57:46 on dist-test-slave-wg14"
I20260620 01:57:46.625028 19838 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:46.660673 19838 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:46.661832 19838 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:46.662182 19838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:46.663866 19838 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:46.666293 19838 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:46.666437 19838 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:46.666569 19838 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:46.666666 19838 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:46.724496 19838 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:33493
I20260620 01:57:46.724560 19965 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:33493 every 8 connection(s)
I20260620 01:57:46.726429 19838 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:57:46.727584 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 19838
I20260620 01:57:46.727841 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:57:46.776229 19966 heartbeater.cc:344] Connected to a master server at 127.20.1.188:39459
I20260620 01:57:46.776757 19966 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:46.778091 19966 heartbeater.cc:507] Master 127.20.1.188:39459 requested a full tablet report, sending...
I20260620 01:57:46.781476 19741 ts_manager.cc:194] Registered new tserver with Master: 7fa3c745ff6c4a3ea05064d339d1c4a9 (127.20.1.129:33493)
I20260620 01:57:46.784996 19970 heartbeater.cc:344] Connected to a master server at 127.20.1.189:45367
I20260620 01:57:46.785212 19970 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:46.785784 19970 heartbeater.cc:507] Master 127.20.1.189:45367 requested a full tablet report, sending...
I20260620 01:57:46.788440 19668 ts_manager.cc:194] Registered new tserver with Master: 7fa3c745ff6c4a3ea05064d339d1c4a9 (127.20.1.129:33493)
I20260620 01:57:46.790330 19967 heartbeater.cc:344] Connected to a master server at 127.20.1.190:33935
I20260620 01:57:46.790570 19967 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:46.791196 19967 heartbeater.cc:507] Master 127.20.1.190:33935 requested a full tablet report, sending...
I20260620 01:57:46.793486 19594 ts_manager.cc:194] Registered new tserver with Master: 7fa3c745ff6c4a3ea05064d339d1c4a9 (127.20.1.129:33493)
I20260620 01:57:46.796319 19594 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:40591
I20260620 01:57:46.804689 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:57:46.808151 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:33439
--webserver_interface=127.20.1.187
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459,127.20.1.187:33439 with env {}
W20260620 01:57:47.318276 19976 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:47.318794 19976 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:47.318890 19976 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:47.333401 19976 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:47.333540 19976 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:47.333596 19976 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:47.333639 19976 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:47.346062 19976 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459,127.20.1.187:33439
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:33439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.20.1.187
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:47.348359 19976 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:47.351011 19976 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260620 01:57:47.366524 19976 server_base.cc:1061] running on GCE node
W20260620 01:57:47.366509 19984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:47.368943 19982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:47.367842 19981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:47.373799 19976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:47.376696 19976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:47.378145 19976 hybrid_clock.cc:648] HybridClock initialized: now 1781920667378069 us; error 58 us; skew 500 ppm
I20260620 01:57:47.378755 19976 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:47.382357 19976 webserver.cc:533] Webserver started at http://127.20.1.187:33643/ using document root <none> and password file <none>
I20260620 01:57:47.384145 19976 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:47.384372 19976 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:47.384987 19976 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:47.389147 19976 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/data/instance:
uuid: "2914eb5d014447f8baccd643337a2441"
format_stamp: "Formatted at 2026-06-20 01:57:47 on dist-test-slave-wg14"
I20260620 01:57:47.390460 19976 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/wal/instance:
uuid: "2914eb5d014447f8baccd643337a2441"
format_stamp: "Formatted at 2026-06-20 01:57:47 on dist-test-slave-wg14"
I20260620 01:57:47.402738 19976 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.008s sys 0.004s
I20260620 01:57:47.408982 19990 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:47.411620 19976 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260620 01:57:47.411901 19976 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "2914eb5d014447f8baccd643337a2441"
format_stamp: "Formatted at 2026-06-20 01:57:47 on dist-test-slave-wg14"
I20260620 01:57:47.412189 19976 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:47.440760 19976 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:47.442768 19976 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:47.443208 19976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:47.475699 20041 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:33439 every 8 connection(s)
I20260620 01:57:47.475724 19976 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:33439
I20260620 01:57:47.478188 19976 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 01:57:47.479101 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 19976
I20260620 01:57:47.485831 20042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:57:47.496526 20042 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:47.514631 20042 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:47.525840 20042 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:47.535475 20042 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:47.549196 20042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: Bootstrap starting.
I20260620 01:57:47.553582 20042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: Neither blocks nor log segments found. Creating new log.
I20260620 01:57:47.555778 20042 log.cc:826] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:47.560629 20042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: No bootstrap required, opened a new log
I20260620 01:57:47.566548 20042 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } }
I20260620 01:57:47.566937 20042 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:57:47.567040 20042 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2914eb5d014447f8baccd643337a2441, State: Initialized, Role: FOLLOWER
I20260620 01:57:47.568179 20042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } }
I20260620 01:57:47.569458 20051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } } }
I20260620 01:57:47.569834 20051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:47.570912 20042 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:57:47.586931 20062 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:57:47.587359 20062 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:57:47.609886 19976 master_runner.cc:429] Detected that this master 2914eb5d014447f8baccd643337a2441 is joining an existing cluster
I20260620 01:57:47.610072 19976 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260620 01:57:47.611274 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 19838
I20260620 01:57:47.611878 19976 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 01:57:47.612464 19976 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 01:57:47.612748 19976 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: stopping tablet replica
I20260620 01:57:47.615563 19976 master_runner.cc:434] Initiating AddMaster RPC to add 127.20.1.187:33439
I20260620 01:57:47.624400 19961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:47.720897 19594 catalog_manager.cc:7201] Initiating ChangeConfig request to add master 127.20.1.187:33439: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2914eb5d014447f8baccd643337a2441"
member_type: NON_VOTER
last_known_addr {
host: "127.20.1.187"
port: 33439
}
attrs {
promote: true
}
}
dest_uuid: "95e50c60221447b79e64a3c0ceac162f"
cas_config_opid_index: -1
I20260620 01:57:47.723058 19594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: true } }
I20260620 01:57:47.727986 19678 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 1 FOLLOWER]: Refusing update from remote peer 95e50c60221447b79e64a3c0ceac162f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260620 01:57:47.728992 19584 consensus_peers.cc:597] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f -> Peer 2914eb5d014447f8baccd643337a2441 (127.20.1.187:33439): Couldn't send request to peer 2914eb5d014447f8baccd643337a2441. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260620 01:57:47.729326 19820 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [LEADER]: Connected to new peer: Peer: permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:57:47.731195 19751 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 1 FOLLOWER]: Refusing update from remote peer 95e50c60221447b79e64a3c0ceac162f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 01:57:47.733151 19820 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 01:57:47.743801 20076 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2914eb5d014447f8baccd643337a2441 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: true } } }
I20260620 01:57:47.745277 19678 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2914eb5d014447f8baccd643337a2441 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: true } } }
I20260620 01:57:47.755112 20079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: true } } }
I20260620 01:57:47.756284 20079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:47.756839 20076 catalog_manager.cc:7205] Successfully completed master ChangeConfig request to add master 127.20.1.187:33439
I20260620 01:57:47.758355 19976 master.cc:562] Master@127.20.1.187:33439 shutting down...
I20260620 01:57:47.768114 20078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: true } } }
I20260620 01:57:47.769070 20078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:47.773180 20076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: true } } }
I20260620 01:57:47.774119 20076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:47.775645 20076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: true } } }
I20260620 01:57:47.776758 20076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:47.801081 19751 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2914eb5d014447f8baccd643337a2441 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: true } } }
I20260620 01:57:47.811933 19976 master.cc:584] Master@127.20.1.187:33439 shutdown complete.
I20260620 01:57:47.811779 20083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: true } } }
I20260620 01:57:47.812163 19976 master_runner.cc:306] Clearing existing system tablet
I20260620 01:57:47.812443 20083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:47.812644 19976 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:47.812793 19976 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:47.817855 19967 heartbeater.cc:499] Master 127.20.1.190:33935 was elected leader, sending a full tablet report...
I20260620 01:57:47.823741 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 19838
I20260620 01:57:47.839593 19976 fs_manager.cc:714] Time spent opening directory manager: real 0.025s user 0.002s sys 0.001s
I20260620 01:57:47.841496 20085 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:47.841969 19976 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:47.842303 19976 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 01:57:47.842437 19976 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "2914eb5d014447f8baccd643337a2441"
format_stamp: "Formatted at 2026-06-20 01:57:47 on dist-test-slave-wg14"
I20260620 01:57:47.846269 19976 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 01:57:47.850883 19655 connection.cc:573] server connection from 127.20.1.129:42347 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:57:47.852674 19976 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 01:57:47.852671 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:33493
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459
--webserver_port=42387
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459,127.20.1.187:33439 with env {}
I20260620 01:57:47.852917 19976 log.cc:1199] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: Deleting WAL directory at /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260620 01:57:47.853524 19976 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: Deleting consensus metadata
I20260620 01:57:47.853807 19976 master_runner.cc:316] Copying system tablet from 127.20.1.190:33935
I20260620 01:57:47.857023 19976 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: tablet copy: Beginning tablet copy session from remote peer at address 127.20.1.190:33935
I20260620 01:57:47.871978 19617 tablet_copy_service.cc:140] P 95e50c60221447b79e64a3c0ceac162f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2914eb5d014447f8baccd643337a2441 ({username='slave'} at 127.0.0.1:50878)
I20260620 01:57:47.872339 19617 tablet_copy_service.cc:161] P 95e50c60221447b79e64a3c0ceac162f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2914eb5d014447f8baccd643337a2441 at {username='slave'} at 127.0.0.1:50878: session id = 2914eb5d014447f8baccd643337a2441-00000000000000000000000000000000
I20260620 01:57:47.878346 19617 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 01:57:47.881795 19976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 01:57:47.889977 19976 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: tablet copy: Starting download of 0 data blocks...
I20260620 01:57:47.890872 19976 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: tablet copy: Starting download of 1 WAL segments...
I20260620 01:57:47.898898 19976 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 01:57:47.904035 19617 tablet_copy_service.cc:342] P 95e50c60221447b79e64a3c0ceac162f: Request end of tablet copy session 2914eb5d014447f8baccd643337a2441-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:50878
I20260620 01:57:47.904382 19617 tablet_copy_service.cc:434] P 95e50c60221447b79e64a3c0ceac162f: ending tablet copy session 2914eb5d014447f8baccd643337a2441-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2914eb5d014447f8baccd643337a2441
W20260620 01:57:47.916857 19976 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260620 01:57:47.923758 19976 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:47.946309 20093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:47.951926 20094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:47.953645 20096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:47.957357 19976 server_base.cc:1061] running on GCE node
I20260620 01:57:47.957978 19976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:47.959337 19976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:47.962850 19976 hybrid_clock.cc:648] HybridClock initialized: now 1781920667962704 us; error 77 us; skew 500 ppm
I20260620 01:57:47.963435 19976 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:47.966886 19976 webserver.cc:533] Webserver started at http://127.20.1.187:38659/ using document root <none> and password file <none>
I20260620 01:57:47.967753 19976 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:47.968073 19976 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:47.972079 19976 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 01:57:47.975404 20102 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:47.976255 19976 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 01:57:47.976591 19976 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "2914eb5d014447f8baccd643337a2441"
format_stamp: "Formatted at 2026-06-20 01:57:47 on dist-test-slave-wg14"
I20260620 01:57:47.976943 19976 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:48.004940 19976 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:48.006131 19976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:48.064815 19976 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:33439
I20260620 01:57:48.064862 20154 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:33439 every 8 connection(s)
I20260620 01:57:48.066390 19976 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 01:57:48.072779 20155 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:57:48.077483 20155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: Bootstrap starting.
I20260620 01:57:48.108331 20155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:57:48.108860 20155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: Bootstrap complete.
I20260620 01:57:48.109979 20155 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: true } }
I20260620 01:57:48.110246 20155 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2914eb5d014447f8baccd643337a2441, State: Initialized, Role: LEARNER
I20260620 01:57:48.110534 20155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: true } }
I20260620 01:57:48.111920 20158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: true } } }
I20260620 01:57:48.112295 20155 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:48.112453 20158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [sys.catalog]: This master's current role is: LEARNER
I20260620 01:57:48.142441 20169 catalog_manager.cc:1294] Loaded cluster ID: 8fe3ea9f935e40efbc81cc6dfa54c5a2
I20260620 01:57:48.143193 20169 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: loading cluster ID for follower catalog manager: success
I20260620 01:57:48.150722 20169 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: acquiring CA information for follower catalog manager: success
I20260620 01:57:48.162406 20169 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:57:48.373260 20130 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260620 01:57:48.374850 20158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95e50c60221447b79e64a3c0ceac162f. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: true } } }
I20260620 01:57:48.375257 20158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [sys.catalog]: This master's current role is: LEARNER
W20260620 01:57:48.522948 20092 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:48.523470 20092 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:48.523594 20092 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:48.536675 20092 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:48.536909 20092 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:48.551209 20092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:33493
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=42387
--tserver_master_addrs=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459,127.20.1.187:33439
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:48.553531 20092 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:48.556535 20092 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:48.571707 20176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:48.571825 20177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:48.572355 20179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:48.574751 20092 server_base.cc:1061] running on GCE node
I20260620 01:57:48.575697 20092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:48.577328 20092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:48.578791 20092 hybrid_clock.cc:648] HybridClock initialized: now 1781920668578702 us; error 72 us; skew 500 ppm
I20260620 01:57:48.579504 20092 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:48.583344 20092 webserver.cc:533] Webserver started at http://127.20.1.129:42387/ using document root <none> and password file <none>
I20260620 01:57:48.584476 20092 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:48.584683 20092 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:48.593856 20092 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.006s sys 0.002s
I20260620 01:57:48.601982 20185 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:48.604928 20092 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.003s
I20260620 01:57:48.605270 20092 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "7fa3c745ff6c4a3ea05064d339d1c4a9"
format_stamp: "Formatted at 2026-06-20 01:57:46 on dist-test-slave-wg14"
I20260620 01:57:48.606345 20092 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:48.659878 20092 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:48.661026 20092 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:48.661378 20092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:48.662601 20092 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:48.665247 20092 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:48.665426 20092 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:48.665601 20092 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:48.665664 20092 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:48.716449 20092 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:33493
I20260620 01:57:48.716584 20297 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:33493 every 8 connection(s)
I20260620 01:57:48.718590 20092 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:57:48.729094 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 20092
I20260620 01:57:48.768229 19820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: true } } }
I20260620 01:57:48.772601 19820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:48.780341 20301 heartbeater.cc:344] Connected to a master server at 127.20.1.189:45367
I20260620 01:57:48.781308 20301 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:48.782974 20301 heartbeater.cc:507] Master 127.20.1.189:45367 requested a full tablet report, sending...
I20260620 01:57:48.785436 20298 heartbeater.cc:344] Connected to a master server at 127.20.1.187:33439
I20260620 01:57:48.785760 20298 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:48.786854 20298 heartbeater.cc:507] Master 127.20.1.187:33439 requested a full tablet report, sending...
I20260620 01:57:48.787348 19668 ts_manager.cc:194] Re-registered known tserver with Master: 7fa3c745ff6c4a3ea05064d339d1c4a9 (127.20.1.129:33493)
I20260620 01:57:48.790428 20120 ts_manager.cc:194] Registered new tserver with Master: 7fa3c745ff6c4a3ea05064d339d1c4a9 (127.20.1.129:33493)
I20260620 01:57:48.790866 20300 heartbeater.cc:344] Connected to a master server at 127.20.1.190:33935
I20260620 01:57:48.791114 20300 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:48.791940 20300 heartbeater.cc:507] Master 127.20.1.190:33935 requested a full tablet report, sending...
I20260620 01:57:48.793879 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:57:48.795786 19594 ts_manager.cc:194] Re-registered known tserver with Master: 7fa3c745ff6c4a3ea05064d339d1c4a9 (127.20.1.129:33493)
I20260620 01:57:48.797416 19594 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:42489
I20260620 01:57:48.807325 20299 heartbeater.cc:344] Connected to a master server at 127.20.1.188:39459
I20260620 01:57:48.807595 20299 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:48.807955 20313 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f: attempting to promote NON_VOTER 2914eb5d014447f8baccd643337a2441 to VOTER
I20260620 01:57:48.808122 20299 heartbeater.cc:507] Master 127.20.1.188:39459 requested a full tablet report, sending...
I20260620 01:57:48.810427 20313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } }
I20260620 01:57:48.811679 19741 ts_manager.cc:194] Re-registered known tserver with Master: 7fa3c745ff6c4a3ea05064d339d1c4a9 (127.20.1.129:33493)
I20260620 01:57:48.815322 20130 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 1 LEARNER]: Refusing update from remote peer 95e50c60221447b79e64a3c0ceac162f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 01:57:48.815490 19751 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 1 FOLLOWER]: Refusing update from remote peer 95e50c60221447b79e64a3c0ceac162f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 01:57:48.816250 19678 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 1 FOLLOWER]: Refusing update from remote peer 95e50c60221447b79e64a3c0ceac162f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 01:57:48.816664 20312 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 01:57:48.817801 20316 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:57:48.818464 20313 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [LEADER]: Connected to new peer: Peer: permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:57:48.820935 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:57:48.821327 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 19566
I20260620 01:57:48.823088 19627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:48.973934 19678 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2914eb5d014447f8baccd643337a2441 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } } }
I20260620 01:57:48.973942 19751 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2914eb5d014447f8baccd643337a2441 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } } }
I20260620 01:57:48.972837 19820 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2914eb5d014447f8baccd643337a2441 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } } }
I20260620 01:57:48.979310 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 19566
I20260620 01:57:48.979349 20312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } } }
I20260620 01:57:48.981369 20321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } } }
I20260620 01:57:48.981766 20321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:48.981065 20130 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2914eb5d014447f8baccd643337a2441 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } } }
I20260620 01:57:48.989094 20322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } } }
I20260620 01:57:48.989482 20322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:48.989313 20323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95e50c60221447b79e64a3c0ceac162f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } } }
I20260620 01:57:48.989674 20323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 01:57:49.008438 20107 connection.cc:573] server connection from 127.0.0.1:34946 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 01:57:49.008788 19654 connection.cc:573] server connection from 127.0.0.1:33462 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 01:57:49.009184 19727 connection.cc:573] server connection from 127.0.0.1:50732 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:57:49.009738 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:33935
--webserver_interface=127.20.1.190
--webserver_port=46671
--builtin_ntp_servers=127.20.1.148:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459
--master_addresses=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459,127.20.1.187:33439 with env {}
W20260620 01:57:49.560881 20324 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:49.561349 20324 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:49.561484 20324 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:49.572422 20324 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:49.572580 20324 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:49.572850 20324 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:49.572978 20324 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:49.588779 20324 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459,127.20.1.187:33439
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:33935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=46671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:49.593698 20324 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:49.597167 20324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:49.611038 20330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:49.611044 20329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:49.613059 20332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:49.614591 20324 server_base.cc:1061] running on GCE node
I20260620 01:57:49.616140 20324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:49.619511 20324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:49.620889 20324 hybrid_clock.cc:648] HybridClock initialized: now 1781920669620829 us; error 68 us; skew 500 ppm
I20260620 01:57:49.621490 20324 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:49.625443 20324 webserver.cc:533] Webserver started at http://127.20.1.190:46671/ using document root <none> and password file <none>
I20260620 01:57:49.626510 20324 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:49.626713 20324 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:49.641052 20324 fs_manager.cc:714] Time spent opening directory manager: real 0.010s user 0.005s sys 0.000s
I20260620 01:57:49.646203 20338 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:49.648484 20324 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:57:49.648734 20324 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "95e50c60221447b79e64a3c0ceac162f"
format_stamp: "Formatted at 2026-06-20 01:57:43 on dist-test-slave-wg14"
I20260620 01:57:49.649729 20324 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:49.693745 20324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:49.695379 20324 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:49.696543 20324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:49.731535 20324 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:33935
I20260620 01:57:49.731535 20389 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:33935 every 8 connection(s)
I20260620 01:57:49.733709 20324 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:57:49.743325 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 20324
I20260620 01:57:49.743880 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 20092
I20260620 01:57:49.749536 20390 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:57:49.754475 20390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f: Bootstrap starting.
I20260620 01:57:49.765029 20293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:49.792711 20390 log.cc:826] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:49.813469 20390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:57:49.814236 20390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f: Bootstrap complete.
I20260620 01:57:49.825119 20390 raft_consensus.cc:359] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } }
I20260620 01:57:49.826113 20390 raft_consensus.cc:740] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e50c60221447b79e64a3c0ceac162f, State: Initialized, Role: FOLLOWER
I20260620 01:57:49.827318 20390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } }
I20260620 01:57:49.828872 20396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } } }
I20260620 01:57:49.829500 20396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:49.830336 20390 sys_catalog.cc:565] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:49.859294 20407 catalog_manager.cc:1294] Loaded cluster ID: 8fe3ea9f935e40efbc81cc6dfa54c5a2
I20260620 01:57:49.859989 20407 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f: loading cluster ID for follower catalog manager: success
I20260620 01:57:49.876930 20407 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f: acquiring CA information for follower catalog manager: success
I20260620 01:57:49.882722 20407 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:57:49.892892 20300 heartbeater.cc:499] Master 127.20.1.190:33935 was elected leader, sending a full tablet report...
I20260620 01:57:49.894652 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 20092
I20260620 01:57:49.916222 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:33493
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459
--webserver_port=42387
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459,127.20.1.187:33439 with env {}
I20260620 01:57:49.959928 20324 master_runner.cc:264] No leader master found from master 95e50c60221447b79e64a3c0ceac162f
I20260620 01:57:49.960434 20324 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:57:50.464685 20413 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:50.465392 20413 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:50.465593 20413 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:50.479097 20413 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:50.479472 20413 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:50.487766 20420 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 95e50c60221447b79e64a3c0ceac162f)
I20260620 01:57:50.488168 20420 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } }
I20260620 01:57:50.490851 20419 raft_consensus.cc:493] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 95e50c60221447b79e64a3c0ceac162f)
I20260620 01:57:50.492043 20419 raft_consensus.cc:515] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } }
I20260620 01:57:50.493620 20420 leader_election.cc:290] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 95e50c60221447b79e64a3c0ceac162f (127.20.1.190:33935), 96ef7785a2024d13bf6ab7bb8028039a (127.20.1.189:45367), 0c207000b0c44c5ca7d8575102e1d63a (127.20.1.188:39459)
I20260620 01:57:50.504736 19751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96ef7785a2024d13bf6ab7bb8028039a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "0c207000b0c44c5ca7d8575102e1d63a" is_pre_election: true
I20260620 01:57:50.505199 19751 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96ef7785a2024d13bf6ab7bb8028039a in term 1.
I20260620 01:57:50.511234 20413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:33493
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=42387
--tserver_master_addrs=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459,127.20.1.187:33439
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:50.515551 20413 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:50.519316 20413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260620 01:57:50.529706 20419 leader_election.cc:290] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 95e50c60221447b79e64a3c0ceac162f (127.20.1.190:33935), 0c207000b0c44c5ca7d8575102e1d63a (127.20.1.188:39459), 2914eb5d014447f8baccd643337a2441 (127.20.1.187:33439)
I20260620 01:57:50.540936 20418 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 95e50c60221447b79e64a3c0ceac162f)
I20260620 01:57:50.541368 20418 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } }
I20260620 01:57:50.560235 20365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2914eb5d014447f8baccd643337a2441" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "95e50c60221447b79e64a3c0ceac162f" is_pre_election: true
I20260620 01:57:50.561200 20365 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2914eb5d014447f8baccd643337a2441 in term 1.
I20260620 01:57:50.566780 20418 leader_election.cc:290] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 95e50c60221447b79e64a3c0ceac162f (127.20.1.190:33935), 96ef7785a2024d13bf6ab7bb8028039a (127.20.1.189:45367), 2914eb5d014447f8baccd643337a2441 (127.20.1.187:33439)
I20260620 01:57:50.568223 19678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c207000b0c44c5ca7d8575102e1d63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "96ef7785a2024d13bf6ab7bb8028039a" is_pre_election: true
I20260620 01:57:50.568888 19678 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c207000b0c44c5ca7d8575102e1d63a in term 1.
I20260620 01:57:50.572275 20365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c207000b0c44c5ca7d8575102e1d63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "95e50c60221447b79e64a3c0ceac162f" is_pre_election: true
I20260620 01:57:50.572670 20365 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c207000b0c44c5ca7d8575102e1d63a in term 1.
I20260620 01:57:50.581780 20130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96ef7785a2024d13bf6ab7bb8028039a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2914eb5d014447f8baccd643337a2441" is_pre_election: true
I20260620 01:57:50.582273 20130 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96ef7785a2024d13bf6ab7bb8028039a in term 1.
I20260620 01:57:50.583231 19655 leader_election.cc:304] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0c207000b0c44c5ca7d8575102e1d63a, 2914eb5d014447f8baccd643337a2441, 96ef7785a2024d13bf6ab7bb8028039a; no voters:
I20260620 01:57:50.578577 19728 leader_election.cc:304] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0c207000b0c44c5ca7d8575102e1d63a, 95e50c60221447b79e64a3c0ceac162f, 96ef7785a2024d13bf6ab7bb8028039a; no voters:
I20260620 01:57:50.593745 20418 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:57:50.595093 20418 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 95e50c60221447b79e64a3c0ceac162f)
I20260620 01:57:50.596179 20418 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:57:50.596098 19678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2914eb5d014447f8baccd643337a2441" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "96ef7785a2024d13bf6ab7bb8028039a" is_pre_election: true
I20260620 01:57:50.596460 19678 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2914eb5d014447f8baccd643337a2441 in term 1.
I20260620 01:57:50.597229 20419 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:57:50.597383 20419 raft_consensus.cc:493] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 95e50c60221447b79e64a3c0ceac162f)
I20260620 01:57:50.597478 20419 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:57:50.599385 20106 leader_election.cc:304] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2914eb5d014447f8baccd643337a2441, 95e50c60221447b79e64a3c0ceac162f, 96ef7785a2024d13bf6ab7bb8028039a; no voters:
I20260620 01:57:50.599843 20418 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } }
I20260620 01:57:50.599987 20420 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:57:50.600175 20420 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 95e50c60221447b79e64a3c0ceac162f)
I20260620 01:57:50.600308 20420 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:57:50.600584 20419 raft_consensus.cc:515] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } }
I20260620 01:57:50.601229 20418 leader_election.cc:290] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [CANDIDATE]: Term 2 election: Requested vote from peers 95e50c60221447b79e64a3c0ceac162f (127.20.1.190:33935), 96ef7785a2024d13bf6ab7bb8028039a (127.20.1.189:45367), 2914eb5d014447f8baccd643337a2441 (127.20.1.187:33439)
I20260620 01:57:50.601954 20365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c207000b0c44c5ca7d8575102e1d63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "95e50c60221447b79e64a3c0ceac162f"
I20260620 01:57:50.602319 20365 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:57:50.602351 20364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96ef7785a2024d13bf6ab7bb8028039a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "95e50c60221447b79e64a3c0ceac162f" is_pre_election: true
I20260620 01:57:50.605000 19751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96ef7785a2024d13bf6ab7bb8028039a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "0c207000b0c44c5ca7d8575102e1d63a"
I20260620 01:57:50.605427 19751 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 96ef7785a2024d13bf6ab7bb8028039a in current term 2: Already voted for candidate 0c207000b0c44c5ca7d8575102e1d63a in this term.
I20260620 01:57:50.606565 20419 leader_election.cc:290] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [CANDIDATE]: Term 2 election: Requested vote from peers 95e50c60221447b79e64a3c0ceac162f (127.20.1.190:33935), 0c207000b0c44c5ca7d8575102e1d63a (127.20.1.188:39459), 2914eb5d014447f8baccd643337a2441 (127.20.1.187:33439)
I20260620 01:57:50.606904 19678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c207000b0c44c5ca7d8575102e1d63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "96ef7785a2024d13bf6ab7bb8028039a"
I20260620 01:57:50.609458 19678 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0c207000b0c44c5ca7d8575102e1d63a in current term 2: Already voted for candidate 96ef7785a2024d13bf6ab7bb8028039a in this term.
W20260620 01:57:50.611425 20429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:50.612967 20364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96ef7785a2024d13bf6ab7bb8028039a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "95e50c60221447b79e64a3c0ceac162f"
I20260620 01:57:50.614413 19652 leader_election.cc:304] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 96ef7785a2024d13bf6ab7bb8028039a; no voters: 0c207000b0c44c5ca7d8575102e1d63a, 95e50c60221447b79e64a3c0ceac162f
I20260620 01:57:50.614929 20419 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 01:57:50.615199 20420 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } }
I20260620 01:57:50.615331 20130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96ef7785a2024d13bf6ab7bb8028039a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2914eb5d014447f8baccd643337a2441"
I20260620 01:57:50.616431 20130 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 96ef7785a2024d13bf6ab7bb8028039a in current term 2: Already voted for candidate 2914eb5d014447f8baccd643337a2441 in this term.
W20260620 01:57:50.617652 20437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:50.617830 20128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c207000b0c44c5ca7d8575102e1d63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2914eb5d014447f8baccd643337a2441"
W20260620 01:57:50.618507 20439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:50.618554 20365 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c207000b0c44c5ca7d8575102e1d63a in term 2.
I20260620 01:57:50.616276 20129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c207000b0c44c5ca7d8575102e1d63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2914eb5d014447f8baccd643337a2441" is_pre_election: true
I20260620 01:57:50.618877 20128 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0c207000b0c44c5ca7d8575102e1d63a in current term 2: Already voted for candidate 2914eb5d014447f8baccd643337a2441 in this term.
I20260620 01:57:50.619619 19729 leader_election.cc:304] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0c207000b0c44c5ca7d8575102e1d63a, 95e50c60221447b79e64a3c0ceac162f; no voters: 2914eb5d014447f8baccd643337a2441, 96ef7785a2024d13bf6ab7bb8028039a
I20260620 01:57:50.619735 20420 leader_election.cc:290] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [CANDIDATE]: Term 2 election: Requested vote from peers 95e50c60221447b79e64a3c0ceac162f (127.20.1.190:33935), 96ef7785a2024d13bf6ab7bb8028039a (127.20.1.189:45367), 0c207000b0c44c5ca7d8575102e1d63a (127.20.1.188:39459)
I20260620 01:57:50.620271 20418 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 01:57:50.620700 19678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2914eb5d014447f8baccd643337a2441" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "96ef7785a2024d13bf6ab7bb8028039a"
I20260620 01:57:50.621055 19678 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2914eb5d014447f8baccd643337a2441 in current term 2: Already voted for candidate 96ef7785a2024d13bf6ab7bb8028039a in this term.
I20260620 01:57:50.621690 20365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2914eb5d014447f8baccd643337a2441" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "95e50c60221447b79e64a3c0ceac162f"
I20260620 01:57:50.622154 20365 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2914eb5d014447f8baccd643337a2441 in current term 2: Already voted for candidate 0c207000b0c44c5ca7d8575102e1d63a in this term.
I20260620 01:57:50.622476 20413 server_base.cc:1061] running on GCE node
I20260620 01:57:50.622982 20103 leader_election.cc:304] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2914eb5d014447f8baccd643337a2441; no voters: 95e50c60221447b79e64a3c0ceac162f, 96ef7785a2024d13bf6ab7bb8028039a
I20260620 01:57:50.623335 19751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2914eb5d014447f8baccd643337a2441" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "0c207000b0c44c5ca7d8575102e1d63a" is_pre_election: true
I20260620 01:57:50.623534 20420 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 01:57:50.623906 19751 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2914eb5d014447f8baccd643337a2441 in current term 2: Already voted for candidate 0c207000b0c44c5ca7d8575102e1d63a in this term.
I20260620 01:57:50.623473 19752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2914eb5d014447f8baccd643337a2441" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "0c207000b0c44c5ca7d8575102e1d63a"
I20260620 01:57:50.624164 19752 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2914eb5d014447f8baccd643337a2441 in current term 2: Already voted for candidate 0c207000b0c44c5ca7d8575102e1d63a in this term.
I20260620 01:57:50.624341 20413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:50.626439 20413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:50.627863 20413 hybrid_clock.cc:648] HybridClock initialized: now 1781920670627744 us; error 102 us; skew 500 ppm
I20260620 01:57:50.628685 20413 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:50.633726 20413 webserver.cc:533] Webserver started at http://127.20.1.129:42387/ using document root <none> and password file <none>
I20260620 01:57:50.634804 20413 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:50.634941 20413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:50.642496 20413 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.005s
I20260620 01:57:50.648002 20445 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:50.653077 20413 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.005s sys 0.000s
I20260620 01:57:50.653729 20413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "7fa3c745ff6c4a3ea05064d339d1c4a9"
format_stamp: "Formatted at 2026-06-20 01:57:46 on dist-test-slave-wg14"
I20260620 01:57:50.654840 20413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:50.681798 20413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:50.683081 20413 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:50.683529 20413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:50.685060 20413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:50.688467 20413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:50.688642 20413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:50.688812 20413 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:50.688871 20413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:50.742504 20413 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:33493
I20260620 01:57:50.742560 20563 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:33493 every 8 connection(s)
I20260620 01:57:50.745561 20413 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:57:50.746910 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 20413
I20260620 01:57:50.748404 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 20413
I20260620 01:57:50.791386 20564 heartbeater.cc:344] Connected to a master server at 127.20.1.187:33439
I20260620 01:57:50.792683 20564 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:50.792897 20565 heartbeater.cc:344] Connected to a master server at 127.20.1.188:39459
I20260620 01:57:50.793093 20565 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:50.794981 20564 heartbeater.cc:507] Master 127.20.1.187:33439 requested a full tablet report, sending...
I20260620 01:57:50.795789 20565 heartbeater.cc:507] Master 127.20.1.188:39459 requested a full tablet report, sending...
I20260620 01:57:50.796365 20566 heartbeater.cc:344] Connected to a master server at 127.20.1.190:33935
I20260620 01:57:50.796586 20566 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:50.797333 20566 heartbeater.cc:507] Master 127.20.1.190:33935 requested a full tablet report, sending...
I20260620 01:57:50.798166 19741 ts_manager.cc:194] Re-registered known tserver with Master: 7fa3c745ff6c4a3ea05064d339d1c4a9 (127.20.1.129:33493)
I20260620 01:57:50.798264 20120 ts_manager.cc:194] Re-registered known tserver with Master: 7fa3c745ff6c4a3ea05064d339d1c4a9 (127.20.1.129:33493)
I20260620 01:57:50.799767 20355 ts_manager.cc:194] Registered new tserver with Master: 7fa3c745ff6c4a3ea05064d339d1c4a9 (127.20.1.129:33493)
I20260620 01:57:50.799822 20570 heartbeater.cc:344] Connected to a master server at 127.20.1.189:45367
I20260620 01:57:50.799986 20570 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:50.800606 20570 heartbeater.cc:507] Master 127.20.1.189:45367 requested a full tablet report, sending...
I20260620 01:57:50.802335 19668 ts_manager.cc:194] Re-registered known tserver with Master: 7fa3c745ff6c4a3ea05064d339d1c4a9 (127.20.1.129:33493)
I20260620 01:57:50.805109 20559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:50.913379 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 20413
I20260620 01:57:50.930315 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:33493
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459
--webserver_port=42387
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459,127.20.1.187:33439
--fault_before_start=1.0 with env {}
W20260620 01:57:51.401535 20576 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:51.401928 20576 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:51.402034 20576 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:57:51.402124 20576 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:51.412065 20576 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:51.412293 20576 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:51.423784 20576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:33493
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=42387
--tserver_master_addrs=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459,127.20.1.187:33439
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:51.425827 20576 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:51.428275 20576 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:51.441720 20582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:51.441810 20581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:51.442876 20584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:51.444234 20576 server_base.cc:1061] running on GCE node
I20260620 01:57:51.445385 20576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:51.447557 20576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:51.448982 20576 hybrid_clock.cc:648] HybridClock initialized: now 1781920671448867 us; error 64 us; skew 500 ppm
I20260620 01:57:51.449530 20576 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:51.453495 20576 webserver.cc:533] Webserver started at http://127.20.1.129:42387/ using document root <none> and password file <none>
I20260620 01:57:51.455018 20576 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:51.455302 20576 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:51.463411 20576 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 01:57:51.468950 20590 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:51.471751 20576 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260620 01:57:51.472067 20576 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "7fa3c745ff6c4a3ea05064d339d1c4a9"
format_stamp: "Formatted at 2026-06-20 01:57:46 on dist-test-slave-wg14"
I20260620 01:57:51.473383 20576 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:51.501472 20576 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:51.503437 20576 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:51.503814 20576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:51.505205 20576 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:51.507972 20576 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:51.508138 20576 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:51.508284 20576 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:51.508363 20576 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:57:51.508929 20576 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:57:51.509034 20576 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:57:51.509123 20576 fault_injection.cc:59] injecting fault for kudu (pid 20576): FLAGS_fault_before_start (process will exit)
I20260620 01:57:51.529464 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:33493
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459
--webserver_port=42387
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459,127.20.1.187:33439 with env {}
W20260620 01:57:52.106271 20598 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:52.106773 20598 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:52.106976 20598 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:52.118479 20598 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:52.118721 20598 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:52.136713 20598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:33493
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=42387
--tserver_master_addrs=127.20.1.190:33935,127.20.1.189:45367,127.20.1.188:39459,127.20.1.187:33439
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:52.138917 20598 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:52.141518 20598 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:52.155272 20604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:52.155005 20603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:52.156548 20606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:52.156934 20598 server_base.cc:1061] running on GCE node
I20260620 01:57:52.157836 20598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:52.159437 20598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:52.163074 20598 hybrid_clock.cc:648] HybridClock initialized: now 1781920672162848 us; error 185 us; skew 500 ppm
I20260620 01:57:52.164177 20598 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:52.167883 20598 webserver.cc:533] Webserver started at http://127.20.1.129:42387/ using document root <none> and password file <none>
I20260620 01:57:52.169091 20598 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:52.169250 20598 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:52.184304 20598 fs_manager.cc:714] Time spent opening directory manager: real 0.012s user 0.010s sys 0.000s
I20260620 01:57:52.190508 20612 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:52.192852 20598 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.005s sys 0.001s
I20260620 01:57:52.193233 20598 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "7fa3c745ff6c4a3ea05064d339d1c4a9"
format_stamp: "Formatted at 2026-06-20 01:57:46 on dist-test-slave-wg14"
I20260620 01:57:52.197659 20598 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:52.253429 20598 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:52.254951 20598 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:52.255406 20598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:52.257555 20598 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:52.261358 20598 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:52.261520 20598 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:52.261670 20598 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:52.261730 20598 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:52.319775 20598 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:33493
I20260620 01:57:52.320138 20724 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:33493 every 8 connection(s)
I20260620 01:57:52.321970 20598 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:57:52.323789 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 20598
I20260620 01:57:52.324539 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 20598
I20260620 01:57:52.373135 20725 heartbeater.cc:344] Connected to a master server at 127.20.1.187:33439
I20260620 01:57:52.373744 20725 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:52.379302 20725 heartbeater.cc:507] Master 127.20.1.187:33439 requested a full tablet report, sending...
I20260620 01:57:52.382290 20120 ts_manager.cc:194] Re-registered known tserver with Master: 7fa3c745ff6c4a3ea05064d339d1c4a9 (127.20.1.129:33493)
I20260620 01:57:52.388484 20720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:52.418778 20741 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:57:52.419044 20741 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } }
I20260620 01:57:52.422003 20741 leader_election.cc:290] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95e50c60221447b79e64a3c0ceac162f (127.20.1.190:33935), 96ef7785a2024d13bf6ab7bb8028039a (127.20.1.189:45367), 2914eb5d014447f8baccd643337a2441 (127.20.1.187:33439)
I20260620 01:57:52.423269 20129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c207000b0c44c5ca7d8575102e1d63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2914eb5d014447f8baccd643337a2441" is_pre_election: true
I20260620 01:57:52.423062 20365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c207000b0c44c5ca7d8575102e1d63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "95e50c60221447b79e64a3c0ceac162f" is_pre_election: true
I20260620 01:57:52.423534 19678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c207000b0c44c5ca7d8575102e1d63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "96ef7785a2024d13bf6ab7bb8028039a" is_pre_election: true
I20260620 01:57:52.423731 20365 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c207000b0c44c5ca7d8575102e1d63a in term 2.
I20260620 01:57:52.423744 20129 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c207000b0c44c5ca7d8575102e1d63a in term 2.
I20260620 01:57:52.423857 19678 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c207000b0c44c5ca7d8575102e1d63a in term 2.
I20260620 01:57:52.424950 19726 leader_election.cc:304] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0c207000b0c44c5ca7d8575102e1d63a, 2914eb5d014447f8baccd643337a2441, 95e50c60221447b79e64a3c0ceac162f; no voters:
I20260620 01:57:52.425508 20741 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 01:57:52.425808 20741 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:57:52.426110 20741 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:57:52.429958 20741 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } }
I20260620 01:57:52.431680 20741 leader_election.cc:290] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [CANDIDATE]: Term 3 election: Requested vote from peers 95e50c60221447b79e64a3c0ceac162f (127.20.1.190:33935), 96ef7785a2024d13bf6ab7bb8028039a (127.20.1.189:45367), 2914eb5d014447f8baccd643337a2441 (127.20.1.187:33439)
I20260620 01:57:52.432355 20365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c207000b0c44c5ca7d8575102e1d63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "95e50c60221447b79e64a3c0ceac162f"
I20260620 01:57:52.432403 19678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c207000b0c44c5ca7d8575102e1d63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "96ef7785a2024d13bf6ab7bb8028039a"
I20260620 01:57:52.432760 19678 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:57:52.433010 20365 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:57:52.433182 20129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c207000b0c44c5ca7d8575102e1d63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2914eb5d014447f8baccd643337a2441"
I20260620 01:57:52.433521 20129 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:57:52.437136 20129 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c207000b0c44c5ca7d8575102e1d63a in term 3.
I20260620 01:57:52.437179 20365 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 95e50c60221447b79e64a3c0ceac162f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c207000b0c44c5ca7d8575102e1d63a in term 3.
I20260620 01:57:52.437232 19678 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 96ef7785a2024d13bf6ab7bb8028039a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c207000b0c44c5ca7d8575102e1d63a in term 3.
I20260620 01:57:52.438205 19726 leader_election.cc:304] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0c207000b0c44c5ca7d8575102e1d63a, 95e50c60221447b79e64a3c0ceac162f, 96ef7785a2024d13bf6ab7bb8028039a; no voters:
I20260620 01:57:52.438787 20741 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 3 FOLLOWER]: Leader election won for term 3
I20260620 01:57:52.439496 20741 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [term 3 LEADER]: Becoming Leader. State: Replica: 0c207000b0c44c5ca7d8575102e1d63a, State: Running, Role: LEADER
I20260620 01:57:52.440102 20741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } }
I20260620 01:57:52.442826 20742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c207000b0c44c5ca7d8575102e1d63a. Latest consensus state: current_term: 3 leader_uuid: "0c207000b0c44c5ca7d8575102e1d63a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } } }
I20260620 01:57:52.443287 20742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:52.444273 20744 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:57:52.448310 20744 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:57:52.450166 20744 catalog_manager.cc:1294] Loaded cluster ID: 8fe3ea9f935e40efbc81cc6dfa54c5a2
I20260620 01:57:52.450307 20744 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:57:52.452224 20744 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:57:52.453524 20744 catalog_manager.cc:6080] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a: Loaded TSK: 0
I20260620 01:57:52.454815 20744 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:57:52.505973 20726 heartbeater.cc:344] Connected to a master server at 127.20.1.188:39459
I20260620 01:57:52.506310 20726 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:52.507622 20726 heartbeater.cc:507] Master 127.20.1.188:39459 requested a full tablet report, sending...
I20260620 01:57:52.508594 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 20598
I20260620 01:57:52.511255 19741 ts_manager.cc:194] Re-registered known tserver with Master: 7fa3c745ff6c4a3ea05064d339d1c4a9 (127.20.1.129:33493)
I20260620 01:57:52.513975 19741 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:39285
W20260620 01:57:52.525947 19729 connection.cc:573] server connection from 127.20.1.129:39285 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:57:52.526968 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 20324
W20260620 01:57:52.527549 20737 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:57:52.363027 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:37545 (local address 127.20.1.190:33935)
0620 01:57:52.374762 (+ 11735us) server_negotiation.cc:207] Beginning negotiation
0620 01:57:52.374785 (+ 23us) server_negotiation.cc:400] Waiting for connection header
0620 01:57:52.374821 (+ 36us) server_negotiation.cc:408] Connection header received
0620 01:57:52.375046 (+ 225us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:57:52.375307 (+ 261us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:57:52.375532 (+ 225us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:57:52.375772 (+ 240us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:57:52.384333 (+ 8561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:57:52.385589 (+ 1256us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:57:52.506866 (+121277us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:57:52.507856 (+ 990us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0620 01:57:52.526270 (+ 18414us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:37545: BlockingRecv error: recv got EOF from 127.20.1.129:37545 (error 108)
Metrics: {"server-negotiator.queue_time_us":10610,"thread_start_us":17727,"threads_started":1}
W20260620 01:57:52.527783 20739 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:57:52.364990 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:60769 (local address 127.20.1.189:45367)
0620 01:57:52.386303 (+ 21313us) server_negotiation.cc:207] Beginning negotiation
0620 01:57:52.386311 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:57:52.386340 (+ 29us) server_negotiation.cc:408] Connection header received
0620 01:57:52.386436 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:57:52.386443 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:57:52.386593 (+ 150us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:57:52.386762 (+ 169us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:57:52.388607 (+ 1845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:57:52.390423 (+ 1816us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:57:52.508627 (+118204us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:57:52.511048 (+ 2421us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0620 01:57:52.526511 (+ 15463us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:60769: BlockingRecv error: recv got EOF from 127.20.1.129:60769 (error 108)
Metrics: {"server-negotiator.queue_time_us":19408,"thread_start_us":19064,"threads_started":1}
I20260620 01:57:52.535789 20385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:52.670305 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 20324
I20260620 01:57:52.695973 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 19634
I20260620 01:57:52.697459 19698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:52.840433 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 19634
I20260620 01:57:52.860546 20129 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [term 3 FOLLOWER]: Refusing update from remote peer 0c207000b0c44c5ca7d8575102e1d63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20260620 01:57:52.861729 20742 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0c207000b0c44c5ca7d8575102e1d63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 01:57:52.865180 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 19707
I20260620 01:57:52.865924 20747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c207000b0c44c5ca7d8575102e1d63a. Latest consensus state: current_term: 3 leader_uuid: "0c207000b0c44c5ca7d8575102e1d63a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e50c60221447b79e64a3c0ceac162f" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33935 } } peers { permanent_uuid: "96ef7785a2024d13bf6ab7bb8028039a" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45367 } } peers { permanent_uuid: "0c207000b0c44c5ca7d8575102e1d63a" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 39459 } } peers { permanent_uuid: "2914eb5d014447f8baccd643337a2441" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 33439 } attrs { promote: false } } }
I20260620 01:57:52.866286 20747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2914eb5d014447f8baccd643337a2441 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:52.866715 19772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:53.003340 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 19707
I20260620 01:57:53.024710 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 19976
I20260620 01:57:53.026883 20150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:53.185549 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 19976
2026-06-20T01:57:53Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/1 (9996 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/2
2026-06-20T01:57:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:57:53Z Disabled control of system clock
2026-06-20T01:57:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:57:53Z Disabled control of system clock
2026-06-20T01:57:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:57:53Z Disabled control of system clock
2026-06-20T01:57:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:57:53Z Disabled control of system clock
2026-06-20T01:57:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:57:53Z Disabled control of system clock
I20260620 01:57:53.502779 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:41891
--webserver_interface=127.20.1.190
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633 with env {}
W20260620 01:57:54.022956 20763 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:54.023515 20763 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:54.023718 20763 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:54.036017 20763 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:54.036214 20763 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:54.036288 20763 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:54.037119 20763 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:54.051026 20763 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:41891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:54.053817 20763 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:54.058581 20763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:54.071172 20768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:54.071550 20769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:54.072330 20771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:54.073235 20763 server_base.cc:1061] running on GCE node
I20260620 01:57:54.074281 20763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:54.076696 20763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:54.078553 20763 hybrid_clock.cc:648] HybridClock initialized: now 1781920674078346 us; error 131 us; skew 500 ppm
I20260620 01:57:54.081853 20763 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:54.086803 20763 webserver.cc:533] Webserver started at http://127.20.1.190:39423/ using document root <none> and password file <none>
I20260620 01:57:54.088407 20763 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:54.088667 20763 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:54.089473 20763 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:54.093068 20763 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59"
format_stamp: "Formatted at 2026-06-20 01:57:54 on dist-test-slave-wg14"
I20260620 01:57:54.094653 20763 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59"
format_stamp: "Formatted at 2026-06-20 01:57:54 on dist-test-slave-wg14"
I20260620 01:57:54.111236 20763 fs_manager.cc:696] Time spent creating directory manager: real 0.016s user 0.009s sys 0.004s
I20260620 01:57:54.121048 20777 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:54.123663 20763 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260620 01:57:54.124329 20763 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59"
format_stamp: "Formatted at 2026-06-20 01:57:54 on dist-test-slave-wg14"
I20260620 01:57:54.124751 20763 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:54.167881 20763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:54.170464 20763 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:54.171741 20763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:54.210228 20763 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:41891
I20260620 01:57:54.210258 20828 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:41891 every 8 connection(s)
I20260620 01:57:54.212908 20763 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:57:54.219260 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 20763
I20260620 01:57:54.219530 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:57:54.221427 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:46783
--webserver_interface=127.20.1.189
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633 with env {}
I20260620 01:57:54.223610 20829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:57:54.234449 20829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:54.266677 20829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:57:54.269384 20778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:46783: connect: Connection refused (error 111)
W20260620 01:57:54.272454 20829 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:46783: Network error: Client connection negotiation failed: client connection to 127.20.1.189:46783: connect: Connection refused (error 111)
I20260620 01:57:54.322129 20829 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } attempt: 1
W20260620 01:57:54.326850 20829 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:46783: Network error: Client connection negotiation failed: client connection to 127.20.1.189:46783: connect: Connection refused (error 111)
I20260620 01:57:54.395401 20829 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } attempt: 2
W20260620 01:57:54.398658 20829 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:46783: Network error: Client connection negotiation failed: client connection to 127.20.1.189:46783: connect: Connection refused (error 111)
I20260620 01:57:54.490219 20829 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } attempt: 3
W20260620 01:57:54.492978 20829 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:46783: Network error: Client connection negotiation failed: client connection to 127.20.1.189:46783: connect: Connection refused (error 111)
I20260620 01:57:54.636436 20829 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } attempt: 4
W20260620 01:57:54.640488 20829 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:46783: Network error: Client connection negotiation failed: client connection to 127.20.1.189:46783: connect: Connection refused (error 111)
W20260620 01:57:54.768366 20831 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:54.768831 20831 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:54.768960 20831 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:54.781033 20831 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:54.781183 20831 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:54.781244 20831 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:54.781286 20831 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:54.797102 20831 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:46783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:54.799243 20831 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:54.801714 20831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:54.815331 20842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:54.815243 20840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:54.815480 20839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:54.817024 20831 server_base.cc:1061] running on GCE node
I20260620 01:57:54.819097 20831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:54.822381 20831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:54.826269 20831 hybrid_clock.cc:648] HybridClock initialized: now 1781920674825774 us; error 176 us; skew 500 ppm
I20260620 01:57:54.827023 20831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:54.831115 20831 webserver.cc:533] Webserver started at http://127.20.1.189:35155/ using document root <none> and password file <none>
I20260620 01:57:54.832413 20831 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:54.832607 20831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:54.833029 20831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:54.839073 20831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "a84fe7c4d33744a2b294439d58b17e38"
format_stamp: "Formatted at 2026-06-20 01:57:54 on dist-test-slave-wg14"
I20260620 01:57:54.840941 20831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "a84fe7c4d33744a2b294439d58b17e38"
format_stamp: "Formatted at 2026-06-20 01:57:54 on dist-test-slave-wg14"
I20260620 01:57:54.850514 20831 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20260620 01:57:54.857005 20848 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:54.859567 20831 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.007s sys 0.000s
I20260620 01:57:54.859887 20831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "a84fe7c4d33744a2b294439d58b17e38"
format_stamp: "Formatted at 2026-06-20 01:57:54 on dist-test-slave-wg14"
I20260620 01:57:54.860174 20831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:54.890946 20831 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:54.892251 20831 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:54.892585 20831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:54.922955 20831 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:46783
I20260620 01:57:54.923081 20899 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:46783 every 8 connection(s)
I20260620 01:57:54.925480 20831 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:57:54.932308 20900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:57:54.933586 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 20831
I20260620 01:57:54.933931 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:57:54.935696 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:40633
--webserver_interface=127.20.1.188
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633 with env {}
I20260620 01:57:54.940066 20829 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } attempt: 5
I20260620 01:57:54.942845 20900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:54.965533 20900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:54.967338 20829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:57:54.970794 20829 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40633: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40633: connect: Connection refused (error 111)
I20260620 01:57:54.976868 20900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:57:54.979789 20852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40633: connect: Connection refused (error 111)
W20260620 01:57:54.982939 20900 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40633: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40633: connect: Connection refused (error 111)
I20260620 01:57:55.022151 20829 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } attempt: 1
W20260620 01:57:55.026108 20829 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40633: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40633: connect: Connection refused (error 111)
I20260620 01:57:55.032800 20900 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } attempt: 1
W20260620 01:57:55.036115 20900 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40633: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40633: connect: Connection refused (error 111)
I20260620 01:57:55.094700 20829 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } attempt: 2
W20260620 01:57:55.098305 20829 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40633: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40633: connect: Connection refused (error 111)
I20260620 01:57:55.104465 20900 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } attempt: 2
W20260620 01:57:55.107924 20900 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40633: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40633: connect: Connection refused (error 111)
I20260620 01:57:55.199406 20900 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } attempt: 3
W20260620 01:57:55.203586 20900 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40633: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40633: connect: Connection refused (error 111)
I20260620 01:57:55.204703 20829 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } attempt: 3
W20260620 01:57:55.216572 20829 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40633: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40633: connect: Connection refused (error 111)
I20260620 01:57:55.347034 20900 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } attempt: 4
W20260620 01:57:55.349985 20900 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40633: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40633: connect: Connection refused (error 111)
I20260620 01:57:55.393959 20829 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } attempt: 4
W20260620 01:57:55.397272 20829 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:40633: Network error: Client connection negotiation failed: client connection to 127.20.1.188:40633: connect: Connection refused (error 111)
W20260620 01:57:55.451658 20905 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:55.452080 20905 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:55.452200 20905 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:55.462849 20905 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:55.463003 20905 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:55.463099 20905 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:55.463196 20905 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:55.478353 20905 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:40633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:55.480312 20905 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:55.482664 20905 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:55.497038 20912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:55.497258 20911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:55.498015 20914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:55.500557 20905 server_base.cc:1061] running on GCE node
I20260620 01:57:55.501708 20905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:55.504236 20905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:55.505941 20905 hybrid_clock.cc:648] HybridClock initialized: now 1781920675505720 us; error 129 us; skew 500 ppm
I20260620 01:57:55.506862 20905 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:55.511725 20905 webserver.cc:533] Webserver started at http://127.20.1.188:37935/ using document root <none> and password file <none>
I20260620 01:57:55.512846 20905 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:55.513017 20905 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:55.513453 20905 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:55.516776 20905 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "c2792927ae664f15934b214734230def"
format_stamp: "Formatted at 2026-06-20 01:57:55 on dist-test-slave-wg14"
I20260620 01:57:55.517891 20905 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "c2792927ae664f15934b214734230def"
format_stamp: "Formatted at 2026-06-20 01:57:55 on dist-test-slave-wg14"
I20260620 01:57:55.525992 20905 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s
I20260620 01:57:55.532435 20920 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:55.534750 20905 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260620 01:57:55.535005 20905 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "c2792927ae664f15934b214734230def"
format_stamp: "Formatted at 2026-06-20 01:57:55 on dist-test-slave-wg14"
I20260620 01:57:55.535377 20905 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:55.562049 20905 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:55.563216 20905 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:55.563800 20905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:55.608536 20971 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:40633 every 8 connection(s)
I20260620 01:57:55.608721 20905 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:40633
I20260620 01:57:55.611758 20905 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:57:55.618897 20972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:57:55.621784 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 20905
I20260620 01:57:55.621989 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:57:55.637750 20972 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:55.649297 20900 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } attempt: 5
I20260620 01:57:55.667372 20972 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:55.676605 20829 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } attempt: 5
I20260620 01:57:55.679844 20900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38: Bootstrap starting.
I20260620 01:57:55.679888 20972 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:55.689225 20900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38: Neither blocks nor log segments found. Creating new log.
I20260620 01:57:55.691660 20900 log.cc:826] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:55.700067 20829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59: Bootstrap starting.
I20260620 01:57:55.701269 20900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38: No bootstrap required, opened a new log
I20260620 01:57:55.705924 20972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def: Bootstrap starting.
I20260620 01:57:55.707036 20829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59: Neither blocks nor log segments found. Creating new log.
I20260620 01:57:55.709544 20829 log.cc:826] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:55.711413 20972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def: Neither blocks nor log segments found. Creating new log.
I20260620 01:57:55.711750 20900 raft_consensus.cc:359] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } }
I20260620 01:57:55.712196 20900 raft_consensus.cc:385] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:57:55.712352 20900 raft_consensus.cc:740] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84fe7c4d33744a2b294439d58b17e38, State: Initialized, Role: FOLLOWER
I20260620 01:57:55.713016 20972 log.cc:826] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:55.713307 20900 consensus_queue.cc:260] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } }
I20260620 01:57:55.714294 20829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59: No bootstrap required, opened a new log
I20260620 01:57:55.716097 20983 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } }
I20260620 01:57:55.719336 20983 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:55.718082 20900 sys_catalog.cc:565] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:55.721421 20972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def: No bootstrap required, opened a new log
I20260620 01:57:55.727669 20829 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } }
I20260620 01:57:55.728473 20829 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:57:55.728698 20829 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c7b5ff6ac6a43d48fe43a082e5dcc59, State: Initialized, Role: FOLLOWER
I20260620 01:57:55.730106 20972 raft_consensus.cc:359] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } }
I20260620 01:57:55.730574 20972 raft_consensus.cc:385] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:57:55.730317 20829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } }
I20260620 01:57:55.730772 20972 raft_consensus.cc:740] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2792927ae664f15934b214734230def, State: Initialized, Role: FOLLOWER
I20260620 01:57:55.731876 20972 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } }
I20260620 01:57:55.732574 20988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } }
I20260620 01:57:55.733147 20988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:55.734540 20829 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:55.738096 20972 sys_catalog.cc:565] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:55.744787 20990 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } }
I20260620 01:57:55.745477 20990 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:55.762318 21008 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 01:57:55.767781 20905 master_runner.cc:187] Error getting master registration for 127.20.1.190:41891: OK, instance_id { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" instance_seqno: 1781920674198045 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:57:55.778553 20905 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:57:55.772778 20831 master_runner.cc:187] Error getting master registration for 127.20.1.190:41891: OK, instance_id { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" instance_seqno: 1781920674198045 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:57:55.783867 20831 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:57:55.788538 21011 catalog_manager.cc:1593] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:57:55.788909 21011 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 01:57:55.802937 21019 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:57:55.803207 21019 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:57:55.806922 20763 master_runner.cc:264] No leader master found from master 8c7b5ff6ac6a43d48fe43a082e5dcc59
I20260620 01:57:55.807168 20763 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:57:55.823990 20926 tablet.cc:2406] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 01:57:55.896670 20854 tablet.cc:2406] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 01:57:55.928571 20783 tablet.cc:2406] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:57:55.946271 20988 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:57:55.946693 20988 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } }
I20260620 01:57:55.948896 20988 leader_election.cc:290] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a84fe7c4d33744a2b294439d58b17e38 (127.20.1.189:46783), c2792927ae664f15934b214734230def (127.20.1.188:40633)
I20260620 01:57:55.949994 20875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a84fe7c4d33744a2b294439d58b17e38" is_pre_election: true
I20260620 01:57:55.950204 20947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2792927ae664f15934b214734230def" is_pre_election: true
I20260620 01:57:55.950618 20875 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c7b5ff6ac6a43d48fe43a082e5dcc59 in term 0.
I20260620 01:57:55.950747 20947 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c7b5ff6ac6a43d48fe43a082e5dcc59 in term 0.
I20260620 01:57:55.951617 20781 leader_election.cc:304] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c7b5ff6ac6a43d48fe43a082e5dcc59, c2792927ae664f15934b214734230def; no voters:
I20260620 01:57:55.952296 20988 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:57:55.952742 20988 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:57:55.953013 20988 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:57:55.960273 20988 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } }
I20260620 01:57:55.963711 20875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a84fe7c4d33744a2b294439d58b17e38"
I20260620 01:57:55.964368 20875 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:57:55.969081 20875 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c7b5ff6ac6a43d48fe43a082e5dcc59 in term 1.
I20260620 01:57:55.969893 20778 leader_election.cc:304] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c7b5ff6ac6a43d48fe43a082e5dcc59, a84fe7c4d33744a2b294439d58b17e38; no voters:
I20260620 01:57:55.970129 20988 leader_election.cc:290] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [CANDIDATE]: Term 1 election: Requested vote from peers a84fe7c4d33744a2b294439d58b17e38 (127.20.1.189:46783), c2792927ae664f15934b214734230def (127.20.1.188:40633)
I20260620 01:57:55.971385 20947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2792927ae664f15934b214734230def"
I20260620 01:57:55.971704 20947 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:57:55.977643 20947 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c7b5ff6ac6a43d48fe43a082e5dcc59 in term 1.
I20260620 01:57:55.979924 20988 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:57:55.980394 20988 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 1 LEADER]: Becoming Leader. State: Replica: 8c7b5ff6ac6a43d48fe43a082e5dcc59, State: Running, Role: LEADER
I20260620 01:57:55.984377 20988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } }
I20260620 01:57:56.002310 21021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c7b5ff6ac6a43d48fe43a082e5dcc59. Latest consensus state: current_term: 1 leader_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } }
I20260620 01:57:56.003559 21021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:56.005599 21023 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:57:56.008701 21023 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:57:56.025996 20947 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [term 1 FOLLOWER]: Refusing update from remote peer 8c7b5ff6ac6a43d48fe43a082e5dcc59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:57:56.026978 20875 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 1 FOLLOWER]: Refusing update from remote peer 8c7b5ff6ac6a43d48fe43a082e5dcc59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:57:56.028918 21021 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:57:56.034742 21021 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:57:56.065989 20990 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c7b5ff6ac6a43d48fe43a082e5dcc59. Latest consensus state: current_term: 1 leader_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } }
I20260620 01:57:56.066361 20990 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:56.078498 20990 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } }
I20260620 01:57:56.079116 20990 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:56.082089 20983 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c7b5ff6ac6a43d48fe43a082e5dcc59. Latest consensus state: current_term: 1 leader_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } }
I20260620 01:57:56.082460 20983 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:56.091188 21021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } }
I20260620 01:57:56.091570 21021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:56.091924 21021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } }
I20260620 01:57:56.092155 21021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:56.093251 21023 catalog_manager.cc:1382] Generated new cluster ID: 31ba3c923e1d4e8c916839856c6fb73e
I20260620 01:57:56.093375 21023 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:57:56.105003 20983 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } }
I20260620 01:57:56.105288 20983 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:56.145309 21023 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:57:56.147643 21023 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:57:56.177543 21023 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59: Generated new TSK 0
I20260620 01:57:56.178975 21023 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:57:56.217128 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:57:56.770264 21008 catalog_manager.cc:1294] Loaded cluster ID: 31ba3c923e1d4e8c916839856c6fb73e
I20260620 01:57:56.770474 21008 catalog_manager.cc:1587] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def: loading cluster ID for follower catalog manager: success
I20260620 01:57:56.776440 21008 catalog_manager.cc:1609] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def: acquiring CA information for follower catalog manager: success
I20260620 01:57:56.780715 21008 catalog_manager.cc:1637] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:57:56.792735 21011 catalog_manager.cc:1294] Loaded cluster ID: 31ba3c923e1d4e8c916839856c6fb73e
I20260620 01:57:56.793020 21011 catalog_manager.cc:1587] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38: loading cluster ID for follower catalog manager: success
I20260620 01:57:56.798484 21011 catalog_manager.cc:1609] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38: acquiring CA information for follower catalog manager: success
I20260620 01:57:56.802174 21011 catalog_manager.cc:1637] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 01:57:56.819185 21033 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:56.819669 21033 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:56.819830 21033 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:56.831313 21033 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:56.831604 21033 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:56.844883 21033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:56.847424 21033 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:56.851064 21033 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:56.866318 21039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:56.869256 21040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:56.870818 21042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:56.871642 21033 server_base.cc:1061] running on GCE node
I20260620 01:57:56.872871 21033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:56.875348 21033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:56.879004 21033 hybrid_clock.cc:648] HybridClock initialized: now 1781920676878739 us; error 158 us; skew 500 ppm
I20260620 01:57:56.880616 21033 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:56.885550 21033 webserver.cc:533] Webserver started at http://127.20.1.129:38839/ using document root <none> and password file <none>
I20260620 01:57:56.886934 21033 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:56.887184 21033 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:56.887771 21033 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:56.900425 21033 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "d6a686b0caf842a887f2a2bb1003f2b9"
format_stamp: "Formatted at 2026-06-20 01:57:56 on dist-test-slave-wg14"
I20260620 01:57:56.902576 21033 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "d6a686b0caf842a887f2a2bb1003f2b9"
format_stamp: "Formatted at 2026-06-20 01:57:56 on dist-test-slave-wg14"
I20260620 01:57:56.916446 21033 fs_manager.cc:696] Time spent creating directory manager: real 0.013s user 0.010s sys 0.001s
I20260620 01:57:56.923905 21048 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:56.925809 21033 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:57:56.926070 21033 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "d6a686b0caf842a887f2a2bb1003f2b9"
format_stamp: "Formatted at 2026-06-20 01:57:56 on dist-test-slave-wg14"
I20260620 01:57:56.926358 21033 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:56.958770 21033 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:56.959968 21033 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:56.960366 21033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:56.961772 21033 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:56.964639 21033 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:56.964804 21033 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:56.964949 21033 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:56.965049 21033 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:57.025274 21033 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35633
I20260620 01:57:57.025336 21160 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35633 every 8 connection(s)
I20260620 01:57:57.027667 21033 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:57:57.038291 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 21033
I20260620 01:57:57.038669 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:57:57.078912 21162 heartbeater.cc:344] Connected to a master server at 127.20.1.190:41891
I20260620 01:57:57.079537 21162 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:57.079703 21161 heartbeater.cc:344] Connected to a master server at 127.20.1.188:40633
I20260620 01:57:57.079842 21163 heartbeater.cc:344] Connected to a master server at 127.20.1.189:46783
I20260620 01:57:57.079936 21161 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:57.080029 21163 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:57.081434 21161 heartbeater.cc:507] Master 127.20.1.188:40633 requested a full tablet report, sending...
I20260620 01:57:57.081538 21162 heartbeater.cc:507] Master 127.20.1.190:41891 requested a full tablet report, sending...
I20260620 01:57:57.081518 21163 heartbeater.cc:507] Master 127.20.1.189:46783 requested a full tablet report, sending...
I20260620 01:57:57.085481 20937 ts_manager.cc:194] Registered new tserver with Master: d6a686b0caf842a887f2a2bb1003f2b9 (127.20.1.129:35633)
I20260620 01:57:57.085462 20865 ts_manager.cc:194] Registered new tserver with Master: d6a686b0caf842a887f2a2bb1003f2b9 (127.20.1.129:35633)
I20260620 01:57:57.088034 20794 ts_manager.cc:194] Registered new tserver with Master: d6a686b0caf842a887f2a2bb1003f2b9 (127.20.1.129:35633)
I20260620 01:57:57.089919 20794 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:60499
I20260620 01:57:57.104753 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:57:57.108197 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:34719
--webserver_interface=127.20.1.187
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633,127.20.1.187:34719 with env {}
W20260620 01:57:57.610390 21171 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:57.611286 21171 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:57.611806 21171 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:57.623488 21171 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:57.623657 21171 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:57.623746 21171 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:57.623811 21171 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:57.637946 21171 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633,127.20.1.187:34719
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:34719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.20.1.187
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:57.639879 21171 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:57.642330 21171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:57.656065 21176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:57.657956 21177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:57.659369 21179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:57.660517 21171 server_base.cc:1061] running on GCE node
I20260620 01:57:57.662206 21171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:57.665199 21171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:57.668926 21171 hybrid_clock.cc:648] HybridClock initialized: now 1781920677668571 us; error 229 us; skew 500 ppm
I20260620 01:57:57.669541 21171 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:57.672770 21171 webserver.cc:533] Webserver started at http://127.20.1.187:34253/ using document root <none> and password file <none>
I20260620 01:57:57.673869 21171 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:57.674043 21171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:57.674515 21171 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:57:57.679183 21171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/data/instance:
uuid: "c57c8a8dc1804d5f861dc834a4b3bf30"
format_stamp: "Formatted at 2026-06-20 01:57:57 on dist-test-slave-wg14"
I20260620 01:57:57.680476 21171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/wal/instance:
uuid: "c57c8a8dc1804d5f861dc834a4b3bf30"
format_stamp: "Formatted at 2026-06-20 01:57:57 on dist-test-slave-wg14"
I20260620 01:57:57.689906 21171 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260620 01:57:57.695385 21185 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:57.698485 21171 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:57:57.699004 21171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "c57c8a8dc1804d5f861dc834a4b3bf30"
format_stamp: "Formatted at 2026-06-20 01:57:57 on dist-test-slave-wg14"
I20260620 01:57:57.699630 21171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:57.716457 21171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:57.718017 21171 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:57.718698 21171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:57.752744 21171 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:34719
I20260620 01:57:57.752835 21236 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:34719 every 8 connection(s)
I20260620 01:57:57.754720 21171 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 01:57:57.755805 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 21171
I20260620 01:57:57.762794 21237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:57:57.771711 21237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:57.788317 21237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:57.798733 21237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:57.810133 21237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34719 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:57:57.827361 21237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: Bootstrap starting.
I20260620 01:57:57.832319 21237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: Neither blocks nor log segments found. Creating new log.
I20260620 01:57:57.833906 21237 log.cc:826] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: Log is configured to *not* fsync() on all Append() calls
I20260620 01:57:57.839118 21237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: No bootstrap required, opened a new log
I20260620 01:57:57.847023 21237 raft_consensus.cc:359] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34719 } }
I20260620 01:57:57.847529 21237 raft_consensus.cc:385] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:57:57.847630 21237 raft_consensus.cc:740] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c8a8dc1804d5f861dc834a4b3bf30, State: Initialized, Role: FOLLOWER
I20260620 01:57:57.848532 21237 consensus_queue.cc:260] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34719 } }
I20260620 01:57:57.849746 21246 sys_catalog.cc:455] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34719 } } }
I20260620 01:57:57.850121 21246 sys_catalog.cc:458] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:57.851370 21237 sys_catalog.cc:565] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:57:57.881649 21257 catalog_manager.cc:1593] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:57:57.881892 21257 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:57:57.886093 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 21033
I20260620 01:57:57.897181 21171 master_runner.cc:429] Detected that this master c57c8a8dc1804d5f861dc834a4b3bf30 is joining an existing cluster
I20260620 01:57:57.897388 21171 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260620 01:57:57.899487 21171 raft_consensus.cc:2243] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 01:57:57.901510 21171 raft_consensus.cc:2272] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 01:57:57.902395 21171 tablet_replica.cc:333] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: stopping tablet replica
I20260620 01:57:57.904644 21156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:57.905670 21171 master_runner.cc:434] Initiating AddMaster RPC to add 127.20.1.187:34719
I20260620 01:57:57.992352 20794 catalog_manager.cc:7201] Initiating ChangeConfig request to add master 127.20.1.187:34719: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30"
member_type: NON_VOTER
last_known_addr {
host: "127.20.1.187"
port: 34719
}
attrs {
promote: true
}
}
dest_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59"
cas_config_opid_index: -1
I20260620 01:57:57.993938 20794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } }
I20260620 01:57:58.001663 20875 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 1 FOLLOWER]: Refusing update from remote peer 8c7b5ff6ac6a43d48fe43a082e5dcc59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260620 01:57:58.001672 20778 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 -> Peer c57c8a8dc1804d5f861dc834a4b3bf30 (127.20.1.187:34719): Couldn't send request to peer c57c8a8dc1804d5f861dc834a4b3bf30. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260620 01:57:58.004326 21032 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:57:58.006328 20947 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [term 1 FOLLOWER]: Refusing update from remote peer 8c7b5ff6ac6a43d48fe43a082e5dcc59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 01:57:58.008843 21032 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:57:58.026966 21032 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c57c8a8dc1804d5f861dc834a4b3bf30 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:57:58.028173 20874 raft_consensus.cc:2955] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c57c8a8dc1804d5f861dc834a4b3bf30 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:57:58.029012 20947 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c57c8a8dc1804d5f861dc834a4b3bf30 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:57:58.031873 21032 catalog_manager.cc:7205] Successfully completed master ChangeConfig request to add master 127.20.1.187:34719
I20260620 01:57:58.032981 21171 master.cc:562] Master@127.20.1.187:34719 shutting down...
I20260620 01:57:58.050285 21273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:57:58.050841 21273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:58.051558 21274 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:57:58.052073 21274 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:58.054932 21032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:57:58.055609 21032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:58.056300 21032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:57:58.056838 21032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:58.074491 21171 master.cc:584] Master@127.20.1.187:34719 shutdown complete.
I20260620 01:57:58.074664 21171 master_runner.cc:306] Clearing existing system tablet
I20260620 01:57:58.075477 21171 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:58.075714 21171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:58.087730 21171 fs_manager.cc:714] Time spent opening directory manager: real 0.009s user 0.005s sys 0.004s
I20260620 01:57:58.087877 21275 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:57:58.088407 21275 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:57:58.097582 21279 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:58.099858 21171 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:58.100744 21171 fs_manager.cc:730] Time spent opening block manager: real 0.012s user 0.005s sys 0.005s
I20260620 01:57:58.100996 21171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "c57c8a8dc1804d5f861dc834a4b3bf30"
format_stamp: "Formatted at 2026-06-20 01:57:57 on dist-test-slave-wg14"
I20260620 01:57:58.110347 21162 heartbeater.cc:499] Master 127.20.1.190:41891 was elected leader, sending a full tablet report...
I20260620 01:57:58.116880 21171 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 01:57:58.126034 21171 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 01:57:58.126859 21171 log.cc:1199] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: Deleting WAL directory at /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260620 01:57:58.127671 21171 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: Deleting consensus metadata
I20260620 01:57:58.128010 21171 master_runner.cc:316] Copying system tablet from 127.20.1.190:41891
I20260620 01:57:58.134321 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 21033
I20260620 01:57:58.140322 21171 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: tablet copy: Beginning tablet copy session from remote peer at address 127.20.1.190:41891
I20260620 01:57:58.162443 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35633
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633
--webserver_port=38839
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633,127.20.1.187:34719 with env {}
I20260620 01:57:58.166244 20814 tablet_copy_service.cc:140] P 8c7b5ff6ac6a43d48fe43a082e5dcc59: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c57c8a8dc1804d5f861dc834a4b3bf30 ({username='slave'} at 127.0.0.1:58816)
I20260620 01:57:58.166705 20814 tablet_copy_service.cc:161] P 8c7b5ff6ac6a43d48fe43a082e5dcc59: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c57c8a8dc1804d5f861dc834a4b3bf30 at {username='slave'} at 127.0.0.1:58816: session id = c57c8a8dc1804d5f861dc834a4b3bf30-00000000000000000000000000000000
I20260620 01:57:58.171943 20814 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 01:57:58.175917 21171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 01:57:58.184952 21171 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: tablet copy: Starting download of 0 data blocks...
I20260620 01:57:58.185434 21171 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: tablet copy: Starting download of 1 WAL segments...
I20260620 01:57:58.191808 21171 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 01:57:58.199421 20814 tablet_copy_service.cc:342] P 8c7b5ff6ac6a43d48fe43a082e5dcc59: Request end of tablet copy session c57c8a8dc1804d5f861dc834a4b3bf30-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:58816
I20260620 01:57:58.199841 20814 tablet_copy_service.cc:434] P 8c7b5ff6ac6a43d48fe43a082e5dcc59: ending tablet copy session c57c8a8dc1804d5f861dc834a4b3bf30-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c57c8a8dc1804d5f861dc834a4b3bf30
W20260620 01:57:58.212517 21171 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260620 01:57:58.220517 21171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:58.240283 21287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:58.247083 21288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:58.251608 21290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:58.254498 21171 server_base.cc:1061] running on GCE node
I20260620 01:57:58.255785 21171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:58.259186 21171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:58.267299 21171 hybrid_clock.cc:648] HybridClock initialized: now 1781920678266838 us; error 379 us; skew 500 ppm
I20260620 01:57:58.267987 21171 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:58.271070 21171 webserver.cc:533] Webserver started at http://127.20.1.187:43497/ using document root <none> and password file <none>
I20260620 01:57:58.271838 21171 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:58.272009 21171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:58.277064 21171 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260620 01:57:58.281893 21296 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:58.282671 21171 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 01:57:58.282913 21171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "c57c8a8dc1804d5f861dc834a4b3bf30"
format_stamp: "Formatted at 2026-06-20 01:57:57 on dist-test-slave-wg14"
I20260620 01:57:58.283221 21171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:58.358884 21171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:58.360639 21171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:58.443837 21171 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:34719
I20260620 01:57:58.444078 21347 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:34719 every 8 connection(s)
I20260620 01:57:58.445614 21171 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 01:57:58.451392 21348 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:57:58.459116 21348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: Bootstrap starting.
I20260620 01:57:58.491484 21348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:57:58.492180 21348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: Bootstrap complete.
I20260620 01:57:58.493769 21348 raft_consensus.cc:359] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } }
I20260620 01:57:58.494118 21348 raft_consensus.cc:740] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c57c8a8dc1804d5f861dc834a4b3bf30, State: Initialized, Role: LEARNER
I20260620 01:57:58.494721 21348 consensus_queue.cc:260] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } }
I20260620 01:57:58.496680 21351 sys_catalog.cc:455] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:57:58.497211 21351 sys_catalog.cc:458] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [sys.catalog]: This master's current role is: LEARNER
I20260620 01:57:58.497525 21348 sys_catalog.cc:565] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:57:58.513856 21362 catalog_manager.cc:1294] Loaded cluster ID: 31ba3c923e1d4e8c916839856c6fb73e
I20260620 01:57:58.514051 21362 catalog_manager.cc:1587] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: loading cluster ID for follower catalog manager: success
I20260620 01:57:58.520259 21362 catalog_manager.cc:1609] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: acquiring CA information for follower catalog manager: success
I20260620 01:57:58.526522 21362 catalog_manager.cc:1637] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:57:58.545222 21323 raft_consensus.cc:1217] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260620 01:57:58.546418 21351 sys_catalog.cc:455] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c7b5ff6ac6a43d48fe43a082e5dcc59. Latest consensus state: current_term: 1 leader_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:57:58.546782 21351 sys_catalog.cc:458] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [sys.catalog]: This master's current role is: LEARNER
W20260620 01:57:58.776823 21286 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:58.777462 21286 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:58.777644 21286 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:58.789054 21286 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:58.789285 21286 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:57:58.804105 21286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35633
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=38839
--tserver_master_addrs=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633,127.20.1.187:34719
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:58.806603 21286 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:58.809258 21286 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:58.825685 21369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:58.825671 21370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:58.827391 21372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:58.829279 21286 server_base.cc:1061] running on GCE node
I20260620 01:57:58.830838 21286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:58.838210 21286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:58.841789 21286 hybrid_clock.cc:648] HybridClock initialized: now 1781920678841521 us; error 165 us; skew 500 ppm
I20260620 01:57:58.842587 21286 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:58.850229 21286 webserver.cc:533] Webserver started at http://127.20.1.129:38839/ using document root <none> and password file <none>
I20260620 01:57:58.852484 21286 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:58.853441 21286 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:58.863175 21286 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 01:57:58.868645 21378 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:58.872061 21286 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.004s sys 0.002s
I20260620 01:57:58.872342 21286 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "d6a686b0caf842a887f2a2bb1003f2b9"
format_stamp: "Formatted at 2026-06-20 01:57:56 on dist-test-slave-wg14"
I20260620 01:57:58.873409 21286 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:58.897393 21286 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:58.898648 21286 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:58.898993 21286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:58.900424 21286 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:57:58.903012 21286 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:57:58.903163 21286 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:58.903306 21286 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:57:58.903391 21286 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:58.954953 21286 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35633
I20260620 01:57:58.955026 21490 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35633 every 8 connection(s)
I20260620 01:57:58.957257 21286 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:57:58.963410 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 21286
I20260620 01:57:58.999776 21493 heartbeater.cc:344] Connected to a master server at 127.20.1.187:34719
I20260620 01:57:59.001302 21493 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:59.003847 21493 heartbeater.cc:507] Master 127.20.1.187:34719 requested a full tablet report, sending...
I20260620 01:57:59.005246 21491 heartbeater.cc:344] Connected to a master server at 127.20.1.188:40633
I20260620 01:57:59.005457 21491 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:59.006099 21491 heartbeater.cc:507] Master 127.20.1.188:40633 requested a full tablet report, sending...
I20260620 01:57:59.007666 20937 ts_manager.cc:194] Re-registered known tserver with Master: d6a686b0caf842a887f2a2bb1003f2b9 (127.20.1.129:35633)
I20260620 01:57:59.007900 21313 ts_manager.cc:194] Registered new tserver with Master: d6a686b0caf842a887f2a2bb1003f2b9 (127.20.1.129:35633)
I20260620 01:57:59.009186 21494 heartbeater.cc:344] Connected to a master server at 127.20.1.189:46783
I20260620 01:57:59.009379 21494 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:59.009982 21494 heartbeater.cc:507] Master 127.20.1.189:46783 requested a full tablet report, sending...
I20260620 01:57:59.012099 21492 heartbeater.cc:344] Connected to a master server at 127.20.1.190:41891
I20260620 01:57:59.012327 21492 heartbeater.cc:461] Registering TS with master...
I20260620 01:57:59.012519 20865 ts_manager.cc:194] Re-registered known tserver with Master: d6a686b0caf842a887f2a2bb1003f2b9 (127.20.1.129:35633)
I20260620 01:57:59.012751 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:57:59.014683 21492 heartbeater.cc:507] Master 127.20.1.190:41891 requested a full tablet report, sending...
I20260620 01:57:59.016433 20794 ts_manager.cc:194] Re-registered known tserver with Master: d6a686b0caf842a887f2a2bb1003f2b9 (127.20.1.129:35633)
I20260620 01:57:59.017489 20794 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:57589
I20260620 01:57:59.034343 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:57:59.034585 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 20763
I20260620 01:57:59.035971 20824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:57:59.170652 21273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:57:59.171244 21273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: This master's current role is: LEADER
I20260620 01:57:59.173513 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 20763
I20260620 01:57:59.205475 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:41891
--webserver_interface=127.20.1.190
--webserver_port=39423
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633
--master_addresses=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633,127.20.1.187:34719 with env {}
W20260620 01:57:59.761794 21509 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:57:59.762693 21509 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:57:59.762914 21509 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:57:59.775743 21509 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:57:59.775900 21509 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:57:59.775982 21509 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:57:59.776051 21509 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:57:59.794296 21509 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633,127.20.1.187:34719
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:41891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=39423
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:57:59.796430 21509 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:57:59.798920 21509 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:57:59.818547 21515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:59.820017 21517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:57:59.820022 21514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:57:59.822428 21509 server_base.cc:1061] running on GCE node
I20260620 01:57:59.825042 21509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:57:59.827481 21509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:57:59.829095 21509 hybrid_clock.cc:648] HybridClock initialized: now 1781920679828935 us; error 90 us; skew 500 ppm
I20260620 01:57:59.829661 21509 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:57:59.833956 21509 webserver.cc:533] Webserver started at http://127.20.1.190:39423/ using document root <none> and password file <none>
I20260620 01:57:59.836114 21509 fs_manager.cc:362] Metadata directory not provided
I20260620 01:57:59.836360 21509 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:57:59.846130 21509 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.000s sys 0.005s
I20260620 01:57:59.852198 21523 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:57:59.854787 21509 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.000s sys 0.004s
I20260620 01:57:59.855041 21509 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59"
format_stamp: "Formatted at 2026-06-20 01:57:54 on dist-test-slave-wg14"
I20260620 01:57:59.856019 21509 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:57:59.905382 21509 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:57:59.906729 21509 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:57:59.907209 21509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:57:59.945222 21509 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:41891
I20260620 01:57:59.945238 21574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:41891 every 8 connection(s)
I20260620 01:57:59.949823 21509 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:57:59.957068 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 21509
I20260620 01:57:59.957520 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 21286
I20260620 01:57:59.970948 21575 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:57:59.977535 21575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59: Bootstrap starting.
I20260620 01:57:59.992499 21486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:58:00.012388 21575 log.cc:826] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59: Log is configured to *not* fsync() on all Append() calls
I20260620 01:58:00.037295 21575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:58:00.038079 21575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59: Bootstrap complete.
I20260620 01:58:00.051280 21575 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } }
I20260620 01:58:00.052338 21575 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c7b5ff6ac6a43d48fe43a082e5dcc59, State: Initialized, Role: FOLLOWER
I20260620 01:58:00.053558 21575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } }
I20260620 01:58:00.055033 21581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:58:00.056982 21581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:00.066473 21575 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:58:00.088647 21592 catalog_manager.cc:1294] Loaded cluster ID: 31ba3c923e1d4e8c916839856c6fb73e
I20260620 01:58:00.088832 21592 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59: loading cluster ID for follower catalog manager: success
I20260620 01:58:00.098212 21592 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59: acquiring CA information for follower catalog manager: success
I20260620 01:58:00.106717 21592 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:58:00.158694 21492 heartbeater.cc:499] Master 127.20.1.190:41891 was elected leader, sending a full tablet report...
I20260620 01:58:00.180769 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 21286
I20260620 01:58:00.186512 21509 master_runner.cc:264] No leader master found from master 8c7b5ff6ac6a43d48fe43a082e5dcc59
I20260620 01:58:00.186758 21509 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:58:00.208611 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35633
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633
--webserver_port=38839
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633,127.20.1.187:34719 with env {}
W20260620 01:58:00.208928 21595 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:58:00.164485 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:46937 (local address 127.20.1.190:41891)
0620 01:58:00.165294 (+ 809us) server_negotiation.cc:207] Beginning negotiation
0620 01:58:00.165302 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:58:00.178806 (+ 13504us) server_negotiation.cc:408] Connection header received
0620 01:58:00.180585 (+ 1779us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:58:00.180592 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:58:00.180694 (+ 102us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:58:00.180769 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0620 01:58:00.206911 (+ 26142us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:46937: BlockingRecv error: recv got EOF from 127.20.1.129:46937 (error 108)
Metrics: {"server-negotiator.queue_time_us":621}
I20260620 01:58:00.488662 21601 raft_consensus.cc:493] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c7b5ff6ac6a43d48fe43a082e5dcc59)
I20260620 01:58:00.489210 21601 raft_consensus.cc:515] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } }
I20260620 01:58:00.491771 21601 leader_election.cc:290] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c7b5ff6ac6a43d48fe43a082e5dcc59 (127.20.1.190:41891), c2792927ae664f15934b214734230def (127.20.1.188:40633)
I20260620 01:58:00.495625 20947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a84fe7c4d33744a2b294439d58b17e38" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "c2792927ae664f15934b214734230def" is_pre_election: true
I20260620 01:58:00.510434 21550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a84fe7c4d33744a2b294439d58b17e38" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" is_pre_election: true
I20260620 01:58:00.511777 21550 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a84fe7c4d33744a2b294439d58b17e38 in term 1.
I20260620 01:58:00.513294 20849 leader_election.cc:304] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c7b5ff6ac6a43d48fe43a082e5dcc59, a84fe7c4d33744a2b294439d58b17e38; no voters: c2792927ae664f15934b214734230def
I20260620 01:58:00.514233 21601 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:58:00.514693 21601 raft_consensus.cc:493] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c7b5ff6ac6a43d48fe43a082e5dcc59)
I20260620 01:58:00.514881 21601 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:58:00.518805 21601 raft_consensus.cc:515] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } }
I20260620 01:58:00.521003 20947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a84fe7c4d33744a2b294439d58b17e38" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "c2792927ae664f15934b214734230def"
I20260620 01:58:00.521588 21550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a84fe7c4d33744a2b294439d58b17e38" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59"
I20260620 01:58:00.522361 21601 leader_election.cc:290] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [CANDIDATE]: Term 2 election: Requested vote from peers 8c7b5ff6ac6a43d48fe43a082e5dcc59 (127.20.1.190:41891), c2792927ae664f15934b214734230def (127.20.1.188:40633)
I20260620 01:58:00.522493 21550 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:58:00.532613 21550 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a84fe7c4d33744a2b294439d58b17e38 in term 2.
I20260620 01:58:00.533541 20849 leader_election.cc:304] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c7b5ff6ac6a43d48fe43a082e5dcc59, a84fe7c4d33744a2b294439d58b17e38; no voters: c2792927ae664f15934b214734230def
I20260620 01:58:00.534379 21601 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 01:58:00.535524 21601 raft_consensus.cc:697] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 2 LEADER]: Becoming Leader. State: Replica: a84fe7c4d33744a2b294439d58b17e38, State: Running, Role: LEADER
I20260620 01:58:00.536541 21601 consensus_queue.cc:237] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } }
I20260620 01:58:00.546660 21603 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a84fe7c4d33744a2b294439d58b17e38. Latest consensus state: current_term: 2 leader_uuid: "a84fe7c4d33744a2b294439d58b17e38" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:58:00.547358 21603 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:00.548585 21605 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:58:00.552214 21605 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:58:00.553687 21605 catalog_manager.cc:1294] Loaded cluster ID: 31ba3c923e1d4e8c916839856c6fb73e
I20260620 01:58:00.554005 21605 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:58:00.556188 21605 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:58:00.557650 21605 catalog_manager.cc:6080] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38: Loaded TSK: 0
I20260620 01:58:00.559116 21605 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:58:00.674883 21606 raft_consensus.cc:493] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c7b5ff6ac6a43d48fe43a082e5dcc59)
I20260620 01:58:00.675339 21606 raft_consensus.cc:515] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } }
I20260620 01:58:00.681951 20875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2792927ae664f15934b214734230def" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "a84fe7c4d33744a2b294439d58b17e38" is_pre_election: true
I20260620 01:58:00.681849 21606 leader_election.cc:290] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c7b5ff6ac6a43d48fe43a082e5dcc59 (127.20.1.190:41891), a84fe7c4d33744a2b294439d58b17e38 (127.20.1.189:46783)
I20260620 01:58:00.693048 21550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2792927ae664f15934b214734230def" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" is_pre_election: true
I20260620 01:58:00.693625 21550 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c2792927ae664f15934b214734230def in current term 2: Already voted for candidate a84fe7c4d33744a2b294439d58b17e38 in this term.
I20260620 01:58:00.694590 20921 leader_election.cc:304] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2792927ae664f15934b214734230def; no voters: 8c7b5ff6ac6a43d48fe43a082e5dcc59, a84fe7c4d33744a2b294439d58b17e38
I20260620 01:58:00.695695 21606 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:58:00.699265 21606 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 01:58:00.832585 21600 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:58:00.833057 21600 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:58:00.833194 21600 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:58:00.843690 21600 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:58:00.843962 21600 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:58:00.857398 21600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35633
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=38839
--tserver_master_addrs=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633,127.20.1.187:34719
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:58:00.860514 21600 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:58:00.862920 21600 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:58:00.879545 21612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:00.879356 21613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:00.880450 21615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:58:00.880654 21600 server_base.cc:1061] running on GCE node
I20260620 01:58:00.881500 21600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:58:00.883428 21600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:58:00.885180 21600 hybrid_clock.cc:648] HybridClock initialized: now 1781920680884877 us; error 170 us; skew 500 ppm
I20260620 01:58:00.885682 21600 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:58:00.891582 21600 webserver.cc:533] Webserver started at http://127.20.1.129:38839/ using document root <none> and password file <none>
I20260620 01:58:00.893349 21600 fs_manager.cc:362] Metadata directory not provided
I20260620 01:58:00.893610 21600 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:58:00.908563 21600 fs_manager.cc:714] Time spent opening directory manager: real 0.010s user 0.010s sys 0.001s
I20260620 01:58:00.913722 21621 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:00.916409 21600 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:58:00.916706 21600 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "d6a686b0caf842a887f2a2bb1003f2b9"
format_stamp: "Formatted at 2026-06-20 01:57:56 on dist-test-slave-wg14"
I20260620 01:58:00.917689 21600 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:58:00.941447 21600 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:58:00.943285 21600 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:58:00.943753 21600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:58:00.946086 21600 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:58:00.950322 21600 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:58:00.950501 21600 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:00.950637 21600 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:58:00.950695 21600 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:00.958639 21323 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [term 1 LEARNER]: Advancing to term 2
I20260620 01:58:00.961611 21323 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [term 2 LEARNER]: Refusing update from remote peer a84fe7c4d33744a2b294439d58b17e38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260620 01:58:00.962849 21603 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:58:00.969317 21633 sys_catalog.cc:455] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a84fe7c4d33744a2b294439d58b17e38. Latest consensus state: current_term: 2 leader_uuid: "a84fe7c4d33744a2b294439d58b17e38" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:58:00.970407 21633 sys_catalog.cc:458] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [sys.catalog]: This master's current role is: LEARNER
I20260620 01:58:00.973304 21603 raft_consensus.cc:1064] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38: attempting to promote NON_VOTER c57c8a8dc1804d5f861dc834a4b3bf30 to VOTER
W20260620 01:58:00.973829 21603 raft_consensus.cc:1068] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38: Unable to promote non-voter c57c8a8dc1804d5f861dc834a4b3bf30: Illegal state: Leader has not yet committed an operation in its own term
I20260620 01:58:01.019481 21600 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35633
I20260620 01:58:01.019507 21736 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35633 every 8 connection(s)
I20260620 01:58:01.021597 21600 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:58:01.030046 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 21600
I20260620 01:58:01.030431 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 21600
I20260620 01:58:01.060251 21740 heartbeater.cc:344] Connected to a master server at 127.20.1.189:46783
I20260620 01:58:01.060807 21740 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:01.063179 21740 heartbeater.cc:507] Master 127.20.1.189:46783 requested a full tablet report, sending...
I20260620 01:58:01.066880 21739 heartbeater.cc:344] Connected to a master server at 127.20.1.187:34719
I20260620 01:58:01.067363 21739 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:01.067646 20865 ts_manager.cc:194] Re-registered known tserver with Master: d6a686b0caf842a887f2a2bb1003f2b9 (127.20.1.129:35633)
I20260620 01:58:01.067987 21732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:58:01.072573 20865 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:53295
I20260620 01:58:01.075582 21550 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 2 FOLLOWER]: Refusing update from remote peer a84fe7c4d33744a2b294439d58b17e38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260620 01:58:01.076722 21603 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:58:01.081560 21751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a84fe7c4d33744a2b294439d58b17e38. Latest consensus state: current_term: 2 leader_uuid: "a84fe7c4d33744a2b294439d58b17e38" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:58:01.082136 21751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:01.086261 20947 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [term 2 FOLLOWER]: Refusing update from remote peer a84fe7c4d33744a2b294439d58b17e38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260620 01:58:01.088619 21603 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 01:58:01.097194 21601 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a84fe7c4d33744a2b294439d58b17e38" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:58:01.097615 21601 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:01.098209 21603 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a84fe7c4d33744a2b294439d58b17e38" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:58:01.098503 21603 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:01.117075 21633 sys_catalog.cc:455] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a84fe7c4d33744a2b294439d58b17e38" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:58:01.117408 21633 sys_catalog.cc:458] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [sys.catalog]: This master's current role is: LEARNER
I20260620 01:58:01.118749 21751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a84fe7c4d33744a2b294439d58b17e38" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:58:01.119031 21751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:01.119697 21601 raft_consensus.cc:1064] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38: attempting to promote NON_VOTER c57c8a8dc1804d5f861dc834a4b3bf30 to VOTER
I20260620 01:58:01.121377 21601 consensus_queue.cc:237] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: false } }
I20260620 01:58:01.126046 21323 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [term 2 LEARNER]: Refusing update from remote peer a84fe7c4d33744a2b294439d58b17e38: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260620 01:58:01.128139 21601 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260620 01:58:01.129334 21550 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 2 FOLLOWER]: Refusing update from remote peer a84fe7c4d33744a2b294439d58b17e38: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260620 01:58:01.130406 21754 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260620 01:58:01.138062 21760 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a84fe7c4d33744a2b294439d58b17e38" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:58:01.138538 21760 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:01.139441 21606 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [sys.catalog]: SysCatalogTable state changed. Reason: New leader a84fe7c4d33744a2b294439d58b17e38. Latest consensus state: current_term: 2 leader_uuid: "a84fe7c4d33744a2b294439d58b17e38" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: true } } }
I20260620 01:58:01.139786 21606 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:01.139490 21754 raft_consensus.cc:2955] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, c57c8a8dc1804d5f861dc834a4b3bf30 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: false } } }
I20260620 01:58:01.140369 20946 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [term 2 FOLLOWER]: Refusing update from remote peer a84fe7c4d33744a2b294439d58b17e38: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260620 01:58:01.140908 21550 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, c57c8a8dc1804d5f861dc834a4b3bf30 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: false } } }
I20260620 01:58:01.141806 21603 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 01:58:01.145085 20946 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, c57c8a8dc1804d5f861dc834a4b3bf30 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: false } } }
I20260620 01:58:01.147765 21751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a84fe7c4d33744a2b294439d58b17e38" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: false } } }
I20260620 01:58:01.148116 21751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7b5ff6ac6a43d48fe43a082e5dcc59 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:01.148108 21323 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, c57c8a8dc1804d5f861dc834a4b3bf30 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: false } } }
I20260620 01:58:01.149775 21603 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 2 leader_uuid: "a84fe7c4d33744a2b294439d58b17e38" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: false } } }
I20260620 01:58:01.150098 21603 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:01.150792 21603 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a84fe7c4d33744a2b294439d58b17e38" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: false } } }
I20260620 01:58:01.151109 21603 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:01.151820 21603 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a84fe7c4d33744a2b294439d58b17e38" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: false } } }
I20260620 01:58:01.152055 21603 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:01.152207 21606 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a84fe7c4d33744a2b294439d58b17e38" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: false } } }
I20260620 01:58:01.152563 21606 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2792927ae664f15934b214734230def [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:01.154315 21633 sys_catalog.cc:455] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a84fe7c4d33744a2b294439d58b17e38" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: false } } }
I20260620 01:58:01.154639 21633 sys_catalog.cc:458] T 00000000000000000000000000000000 P c57c8a8dc1804d5f861dc834a4b3bf30 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:01.220486 21738 heartbeater.cc:344] Connected to a master server at 127.20.1.190:41891
I20260620 01:58:01.220840 21738 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:01.221626 21738 heartbeater.cc:507] Master 127.20.1.190:41891 requested a full tablet report, sending...
I20260620 01:58:01.221674 21737 heartbeater.cc:344] Connected to a master server at 127.20.1.188:40633
I20260620 01:58:01.221735 21739 heartbeater.cc:507] Master 127.20.1.187:34719 requested a full tablet report, sending...
I20260620 01:58:01.221853 21737 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:01.222318 21737 heartbeater.cc:507] Master 127.20.1.188:40633 requested a full tablet report, sending...
I20260620 01:58:01.222826 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 21600
I20260620 01:58:01.224851 20936 ts_manager.cc:194] Re-registered known tserver with Master: d6a686b0caf842a887f2a2bb1003f2b9 (127.20.1.129:35633)
I20260620 01:58:01.225370 21540 ts_manager.cc:194] Registered new tserver with Master: d6a686b0caf842a887f2a2bb1003f2b9 (127.20.1.129:35633)
I20260620 01:58:01.225739 21313 ts_manager.cc:194] Re-registered known tserver with Master: d6a686b0caf842a887f2a2bb1003f2b9 (127.20.1.129:35633)
W20260620 01:58:01.241252 20924 connection.cc:573] server connection from 127.20.1.129:47625 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 01:58:01.241250 21298 connection.cc:573] server connection from 127.20.1.129:41265 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 01:58:01.241230 21524 connection.cc:573] server connection from 127.20.1.129:53113 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:58:01.242600 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35633
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633
--webserver_port=38839
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633,127.20.1.187:34719
--fault_before_start=1.0 with env {}
I20260620 01:58:01.607550 21603 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a84fe7c4d33744a2b294439d58b17e38" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8c7b5ff6ac6a43d48fe43a082e5dcc59" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41891 } } peers { permanent_uuid: "a84fe7c4d33744a2b294439d58b17e38" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 46783 } } peers { permanent_uuid: "c2792927ae664f15934b214734230def" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 40633 } } peers { permanent_uuid: "c57c8a8dc1804d5f861dc834a4b3bf30" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34719 } attrs { promote: false } } }
I20260620 01:58:01.607903 21603 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84fe7c4d33744a2b294439d58b17e38 [sys.catalog]: This master's current role is: LEADER
W20260620 01:58:01.707069 21761 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:58:01.707650 21761 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:58:01.707801 21761 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:58:01.707890 21761 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:58:01.718596 21761 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:58:01.718942 21761 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:58:01.732398 21761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35633
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=38839
--tserver_master_addrs=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633,127.20.1.187:34719
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:58:01.734790 21761 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:58:01.737474 21761 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:58:01.753950 21767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:01.753973 21766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:01.755791 21769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:58:01.757598 21761 server_base.cc:1061] running on GCE node
I20260620 01:58:01.758464 21761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:58:01.760140 21761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:58:01.763752 21761 hybrid_clock.cc:648] HybridClock initialized: now 1781920681763498 us; error 126 us; skew 500 ppm
I20260620 01:58:01.764281 21761 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:58:01.767655 21761 webserver.cc:533] Webserver started at http://127.20.1.129:38839/ using document root <none> and password file <none>
I20260620 01:58:01.769105 21761 fs_manager.cc:362] Metadata directory not provided
I20260620 01:58:01.769433 21761 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:58:01.779942 21761 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 01:58:01.785045 21775 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:01.788612 21761 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.003s sys 0.002s
I20260620 01:58:01.788928 21761 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "d6a686b0caf842a887f2a2bb1003f2b9"
format_stamp: "Formatted at 2026-06-20 01:57:56 on dist-test-slave-wg14"
I20260620 01:58:01.790517 21761 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:58:01.822180 21761 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:58:01.823360 21761 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:58:01.823766 21761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:58:01.825311 21761 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:58:01.828846 21761 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:58:01.829039 21761 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:01.829180 21761 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:58:01.829267 21761 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:58:01.829912 21761 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:58:01.830026 21761 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:58:01.830265 21761 fault_injection.cc:59] injecting fault for kudu (pid 21761): FLAGS_fault_before_start (process will exit)
I20260620 01:58:01.853727 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:35633
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633
--webserver_port=38839
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633,127.20.1.187:34719 with env {}
W20260620 01:58:02.417224 21783 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:58:02.417615 21783 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:58:02.417713 21783 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:58:02.427726 21783 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:58:02.427943 21783 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:58:02.439610 21783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42409,127.20.1.149:46369,127.20.1.150:38503,127.20.1.151:36025,127.20.1.152:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:35633
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=38839
--tserver_master_addrs=127.20.1.190:41891,127.20.1.189:46783,127.20.1.188:40633,127.20.1.187:34719
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:58:02.442028 21783 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:58:02.445206 21783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:58:02.458855 21789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:02.458855 21788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:02.460467 21791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:58:02.462052 21783 server_base.cc:1061] running on GCE node
I20260620 01:58:02.462906 21783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:58:02.464718 21783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:58:02.466276 21783 hybrid_clock.cc:648] HybridClock initialized: now 1781920682466039 us; error 166 us; skew 500 ppm
I20260620 01:58:02.466840 21783 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:58:02.470737 21783 webserver.cc:533] Webserver started at http://127.20.1.129:38839/ using document root <none> and password file <none>
I20260620 01:58:02.471956 21783 fs_manager.cc:362] Metadata directory not provided
I20260620 01:58:02.472134 21783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:58:02.480392 21783 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 01:58:02.486387 21797 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:02.488747 21783 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.004s
I20260620 01:58:02.489043 21783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "d6a686b0caf842a887f2a2bb1003f2b9"
format_stamp: "Formatted at 2026-06-20 01:57:56 on dist-test-slave-wg14"
I20260620 01:58:02.490001 21783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:58:02.523248 21783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:58:02.524483 21783 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:58:02.524863 21783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:58:02.526432 21783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:58:02.528965 21783 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:58:02.529124 21783 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:02.529278 21783 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:58:02.529369 21783 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:02.576912 21783 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35633
I20260620 01:58:02.576937 21909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35633 every 8 connection(s)
I20260620 01:58:02.578960 21783 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:58:02.579501 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 21783
I20260620 01:58:02.579990 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 21783
I20260620 01:58:02.629196 21910 heartbeater.cc:344] Connected to a master server at 127.20.1.188:40633
I20260620 01:58:02.629899 21910 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:02.631753 21910 heartbeater.cc:507] Master 127.20.1.188:40633 requested a full tablet report, sending...
I20260620 01:58:02.635268 20936 ts_manager.cc:194] Re-registered known tserver with Master: d6a686b0caf842a887f2a2bb1003f2b9 (127.20.1.129:35633)
I20260620 01:58:02.639914 21911 heartbeater.cc:344] Connected to a master server at 127.20.1.190:41891
I20260620 01:58:02.640194 21905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:58:02.640223 21911 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:02.755203 21911 heartbeater.cc:507] Master 127.20.1.190:41891 requested a full tablet report, sending...
I20260620 01:58:02.758361 21540 ts_manager.cc:194] Re-registered known tserver with Master: d6a686b0caf842a887f2a2bb1003f2b9 (127.20.1.129:35633)
I20260620 01:58:02.760314 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 21783
W20260620 01:58:02.781004 21525 connection.cc:573] server connection from 127.20.1.129:51665 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:58:02.781553 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 21509
W20260620 01:58:02.782877 21923 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:58:02.633161 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:50843 (local address 127.20.1.189:46783)
0620 01:58:02.640223 (+ 7062us) server_negotiation.cc:207] Beginning negotiation
0620 01:58:02.640231 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:58:02.640259 (+ 28us) server_negotiation.cc:408] Connection header received
0620 01:58:02.640356 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:58:02.640362 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:58:02.642806 (+ 2444us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:58:02.643051 (+ 245us) server_negotiation.cc:227] Negotiated authn=SASL
0620 01:58:02.754649 (+111598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:58:02.756538 (+ 1889us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 01:58:02.762336 (+ 5798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 01:58:02.762829 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0620 01:58:02.781785 (+ 18956us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:50843: BlockingRecv error: recv got EOF from 127.20.1.129:50843 (error 108)
Metrics: {"server-negotiator.queue_time_us":6904,"thread_start_us":6785,"threads_started":1}
I20260620 01:58:02.792086 21570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:58:02.931556 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 21509
I20260620 01:58:02.955101 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 20831
I20260620 01:58:02.956929 20895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:58:03.101950 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 20831
W20260620 01:58:03.129251 21298 connection.cc:573] server connection from 127.0.0.1:42452 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:58:03.129750 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 20905
I20260620 01:58:03.131652 20967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:58:03.291481 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 20905
I20260620 01:58:03.319972 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 21171
I20260620 01:58:03.321592 21343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:58:03.490615 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 21171
2026-06-20T01:58:03Z chronyd exiting
2026-06-20T01:58:03Z chronyd exiting
2026-06-20T01:58:03Z chronyd exiting
2026-06-20T01:58:03Z chronyd exiting
2026-06-20T01:58:03Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/2 (10346 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/3
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/security.properties
2026-06-20 01:58:15: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260620 01:58:29.471174 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44173
--webserver_interface=127.20.1.190
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741 with env {}
W20260620 01:58:29.933486 22215 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:58:29.934262 22215 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:58:29.934382 22215 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:58:29.944931 22215 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:58:29.945086 22215 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:58:29.945165 22215 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:58:29.945209 22215 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:58:29.960409 22215 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:58:29.962318 22215 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:58:29.964597 22215 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:58:29.980435 22221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:29.981223 22220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:29.981263 22223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:58:29.981657 22215 server_base.cc:1061] running on GCE node
I20260620 01:58:29.982709 22215 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:58:29.982949 22215 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:58:29.983073 22215 hybrid_clock.cc:648] HybridClock initialized: now 1781920709983043 us; error 0 us; skew 500 ppm
I20260620 01:58:29.983572 22215 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:58:29.990325 22215 webserver.cc:533] Webserver started at http://127.20.1.190:42499/ using document root <none> and password file <none>
I20260620 01:58:29.991396 22215 fs_manager.cc:362] Metadata directory not provided
I20260620 01:58:29.991557 22215 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:58:29.991990 22215 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:58:29.994983 22215 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "8fb73dffb61a47dca206831b250f5348"
format_stamp: "Formatted at 2026-06-20 01:58:29 on dist-test-slave-wg14"
I20260620 01:58:29.996114 22215 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "8fb73dffb61a47dca206831b250f5348"
format_stamp: "Formatted at 2026-06-20 01:58:29 on dist-test-slave-wg14"
I20260620 01:58:30.013659 22215 fs_manager.cc:696] Time spent creating directory manager: real 0.015s user 0.010s sys 0.005s
I20260620 01:58:30.023241 22228 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:30.025671 22215 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s
I20260620 01:58:30.025981 22215 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "8fb73dffb61a47dca206831b250f5348"
format_stamp: "Formatted at 2026-06-20 01:58:29 on dist-test-slave-wg14"
I20260620 01:58:30.026299 22215 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:58:30.087725 22215 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:58:30.089102 22215 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:58:30.089504 22215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:58:30.132908 22215 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44173
I20260620 01:58:30.132885 22279 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44173 every 8 connection(s)
I20260620 01:58:30.134933 22215 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:58:30.143545 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 22215
I20260620 01:58:30.143779 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:58:30.145344 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:40165
--webserver_interface=127.20.1.189
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741 with env {}
I20260620 01:58:30.145953 22280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:58:30.156888 22280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:58:30.186793 22280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:58:30.188756 22232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:40165: connect: Connection refused (error 111)
W20260620 01:58:30.191689 22280 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:40165: Network error: Client connection negotiation failed: client connection to 127.20.1.189:40165: connect: Connection refused (error 111)
I20260620 01:58:30.241693 22280 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } attempt: 1
W20260620 01:58:30.244544 22280 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:40165: Network error: Client connection negotiation failed: client connection to 127.20.1.189:40165: connect: Connection refused (error 111)
I20260620 01:58:30.313347 22280 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } attempt: 2
W20260620 01:58:30.316095 22280 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:40165: Network error: Client connection negotiation failed: client connection to 127.20.1.189:40165: connect: Connection refused (error 111)
I20260620 01:58:30.407624 22280 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } attempt: 3
W20260620 01:58:30.410277 22280 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:40165: Network error: Client connection negotiation failed: client connection to 127.20.1.189:40165: connect: Connection refused (error 111)
I20260620 01:58:30.554113 22280 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } attempt: 4
W20260620 01:58:30.561980 22280 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:40165: Network error: Client connection negotiation failed: client connection to 127.20.1.189:40165: connect: Connection refused (error 111)
W20260620 01:58:30.673138 22282 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:58:30.673712 22282 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:58:30.673812 22282 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:58:30.685024 22282 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:58:30.685176 22282 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:58:30.685246 22282 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:58:30.685297 22282 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:58:30.699645 22282 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:40165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:58:30.701582 22282 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:58:30.703903 22282 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:58:30.715582 22290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:30.715559 22291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:30.716832 22293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:58:30.723902 22282 server_base.cc:1061] running on GCE node
I20260620 01:58:30.726157 22282 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:58:30.727003 22282 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:58:30.727222 22282 hybrid_clock.cc:648] HybridClock initialized: now 1781920710727193 us; error 0 us; skew 500 ppm
I20260620 01:58:30.727769 22282 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:58:30.732034 22282 webserver.cc:533] Webserver started at http://127.20.1.189:43105/ using document root <none> and password file <none>
I20260620 01:58:30.733098 22282 fs_manager.cc:362] Metadata directory not provided
I20260620 01:58:30.733233 22282 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:58:30.733635 22282 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:58:30.740455 22282 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "fcd356f9391c4f2c806fb32328b70cd1"
format_stamp: "Formatted at 2026-06-20 01:58:30 on dist-test-slave-wg14"
I20260620 01:58:30.743314 22282 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "fcd356f9391c4f2c806fb32328b70cd1"
format_stamp: "Formatted at 2026-06-20 01:58:30 on dist-test-slave-wg14"
I20260620 01:58:30.752770 22282 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260620 01:58:30.759644 22298 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:30.761837 22282 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260620 01:58:30.762097 22282 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "fcd356f9391c4f2c806fb32328b70cd1"
format_stamp: "Formatted at 2026-06-20 01:58:30 on dist-test-slave-wg14"
I20260620 01:58:30.762398 22282 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:58:30.822933 22282 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:58:30.825578 22282 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:58:30.826043 22282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:58:30.861339 22280 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } attempt: 5
W20260620 01:58:30.863988 22280 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:40165: Network error: Client connection negotiation failed: client connection to 127.20.1.189:40165: connect: Connection refused (error 111)
I20260620 01:58:30.865152 22349 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:40165 every 8 connection(s)
I20260620 01:58:30.865260 22282 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:40165
I20260620 01:58:30.868101 22282 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:58:30.871658 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 22282
I20260620 01:58:30.871860 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:58:30.872953 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:44741
--webserver_interface=127.20.1.188
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741 with env {}
I20260620 01:58:30.876761 22350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:58:30.884598 22350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:58:30.913465 22350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:58:30.936625 22350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:58:30.939842 22300 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44741: connect: Connection refused (error 111)
W20260620 01:58:30.948040 22350 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44741: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44741: connect: Connection refused (error 111)
I20260620 01:58:30.997781 22350 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } attempt: 1
W20260620 01:58:31.002249 22350 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44741: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44741: connect: Connection refused (error 111)
I20260620 01:58:31.070732 22350 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } attempt: 2
W20260620 01:58:31.073733 22350 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44741: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44741: connect: Connection refused (error 111)
I20260620 01:58:31.165112 22350 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } attempt: 3
W20260620 01:58:31.168188 22350 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44741: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44741: connect: Connection refused (error 111)
I20260620 01:58:31.311534 22350 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } attempt: 4
W20260620 01:58:31.314486 22350 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44741: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44741: connect: Connection refused (error 111)
I20260620 01:58:31.411489 22280 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } attempt: 6
W20260620 01:58:31.422410 22352 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:58:31.422863 22352 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:58:31.422997 22352 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260620 01:58:31.424355 22280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:58:31.428607 22280 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44741: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44741: connect: Connection refused (error 111)
W20260620 01:58:31.437390 22352 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:58:31.437547 22352 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:58:31.437640 22352 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:58:31.437712 22352 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:58:31.450501 22352 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:44741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:58:31.452912 22352 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:58:31.455320 22352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:58:31.469098 22363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:31.469022 22364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:31.469045 22366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:58:31.469635 22352 server_base.cc:1061] running on GCE node
I20260620 01:58:31.472069 22352 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:58:31.472316 22352 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:58:31.472415 22352 hybrid_clock.cc:648] HybridClock initialized: now 1781920711472391 us; error 0 us; skew 500 ppm
I20260620 01:58:31.472806 22352 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:58:31.478467 22352 webserver.cc:533] Webserver started at http://127.20.1.188:33437/ using document root <none> and password file <none>
I20260620 01:58:31.479617 22352 fs_manager.cc:362] Metadata directory not provided
I20260620 01:58:31.479774 22352 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:58:31.480170 22352 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:58:31.480881 22280 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } attempt: 1
I20260620 01:58:31.483911 22352 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "f1694b9e18cc46e7a3736331c8446d71"
format_stamp: "Formatted at 2026-06-20 01:58:31 on dist-test-slave-wg14"
W20260620 01:58:31.484861 22280 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44741: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44741: connect: Connection refused (error 111)
I20260620 01:58:31.485256 22352 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "f1694b9e18cc46e7a3736331c8446d71"
format_stamp: "Formatted at 2026-06-20 01:58:31 on dist-test-slave-wg14"
I20260620 01:58:31.496179 22352 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.010s sys 0.001s
I20260620 01:58:31.502331 22371 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:31.505331 22352 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260620 01:58:31.505650 22352 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "f1694b9e18cc46e7a3736331c8446d71"
format_stamp: "Formatted at 2026-06-20 01:58:31 on dist-test-slave-wg14"
I20260620 01:58:31.505954 22352 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:58:31.537276 22352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:58:31.538558 22352 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:58:31.538897 22352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:58:31.559276 22280 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } attempt: 2
W20260620 01:58:31.567485 22280 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44741: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44741: connect: Connection refused (error 111)
I20260620 01:58:31.583742 22352 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:44741
I20260620 01:58:31.583731 22422 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:44741 every 8 connection(s)
I20260620 01:58:31.585692 22352 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:58:31.588719 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 22352
I20260620 01:58:31.589058 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:58:31.592854 22423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:58:31.610371 22423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:58:31.614012 22350 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } attempt: 5
I20260620 01:58:31.634857 22423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:58:31.657235 22350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1: Bootstrap starting.
I20260620 01:58:31.657639 22423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:58:31.663525 22350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1: Neither blocks nor log segments found. Creating new log.
I20260620 01:58:31.669050 22350 log.cc:826] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1: Log is configured to *not* fsync() on all Append() calls
I20260620 01:58:31.674702 22350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1: No bootstrap required, opened a new log
I20260620 01:58:31.674957 22423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71: Bootstrap starting.
I20260620 01:58:31.680990 22280 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } attempt: 3
I20260620 01:58:31.688441 22423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71: Neither blocks nor log segments found. Creating new log.
I20260620 01:58:31.691000 22350 raft_consensus.cc:359] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } }
I20260620 01:58:31.691581 22423 log.cc:826] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71: Log is configured to *not* fsync() on all Append() calls
I20260620 01:58:31.692625 22350 raft_consensus.cc:385] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:58:31.693019 22350 raft_consensus.cc:740] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd356f9391c4f2c806fb32328b70cd1, State: Initialized, Role: FOLLOWER
I20260620 01:58:31.695242 22350 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } }
I20260620 01:58:31.700600 22432 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } }
I20260620 01:58:31.701321 22432 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:31.702488 22350 sys_catalog.cc:565] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:58:31.704525 22423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71: No bootstrap required, opened a new log
I20260620 01:58:31.713124 22280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348: Bootstrap starting.
I20260620 01:58:31.717239 22423 raft_consensus.cc:359] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } }
I20260620 01:58:31.718020 22423 raft_consensus.cc:385] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:58:31.718276 22423 raft_consensus.cc:740] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1694b9e18cc46e7a3736331c8446d71, State: Initialized, Role: FOLLOWER
I20260620 01:58:31.719684 22423 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } }
I20260620 01:58:31.724583 22443 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } }
I20260620 01:58:31.725567 22443 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:31.726384 22280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348: Neither blocks nor log segments found. Creating new log.
I20260620 01:58:31.727531 22423 sys_catalog.cc:565] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:58:31.730072 22280 log.cc:826] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348: Log is configured to *not* fsync() on all Append() calls
I20260620 01:58:31.730032 22282 master_runner.cc:187] Error getting master registration for 127.20.1.190:44173: OK, instance_id { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" instance_seqno: 1781920710116569 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:58:31.738598 22282 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:58:31.742923 22444 catalog_manager.cc:1593] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:58:31.743896 22444 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:58:31.753582 22280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348: No bootstrap required, opened a new log
I20260620 01:58:31.771817 22280 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } }
I20260620 01:58:31.772634 22280 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:58:31.772890 22280 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fb73dffb61a47dca206831b250f5348, State: Initialized, Role: FOLLOWER
I20260620 01:58:31.774134 22280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } }
I20260620 01:58:31.775784 22458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } }
I20260620 01:58:31.776432 22458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:31.787649 22280 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:58:31.778365 22352 master_runner.cc:187] Error getting master registration for 127.20.1.190:44173: OK, instance_id { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" instance_seqno: 1781920710116569 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:58:31.792172 22352 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:58:31.792526 22377 tablet.cc:2406] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 01:58:31.802538 22457 catalog_manager.cc:1593] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:58:31.802789 22457 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 01:58:31.830258 22304 tablet.cc:2406] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:58:31.833401 22215 master_runner.cc:264] No leader master found from master 8fb73dffb61a47dca206831b250f5348
I20260620 01:58:31.833786 22215 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:58:31.837849 22470 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:58:31.838269 22470 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 01:58:31.844489 22234 tablet.cc:2406] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:58:31.847597 22458 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:58:31.847978 22458 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } }
I20260620 01:58:31.853017 22325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fb73dffb61a47dca206831b250f5348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd356f9391c4f2c806fb32328b70cd1" is_pre_election: true
I20260620 01:58:31.853797 22325 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fb73dffb61a47dca206831b250f5348 in term 0.
I20260620 01:58:31.855064 22232 leader_election.cc:304] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fb73dffb61a47dca206831b250f5348, fcd356f9391c4f2c806fb32328b70cd1; no voters:
I20260620 01:58:31.856123 22398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fb73dffb61a47dca206831b250f5348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1694b9e18cc46e7a3736331c8446d71" is_pre_election: true
I20260620 01:58:31.857151 22398 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fb73dffb61a47dca206831b250f5348 in term 0.
I20260620 01:58:31.858669 22458 leader_election.cc:290] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcd356f9391c4f2c806fb32328b70cd1 (127.20.1.189:40165), f1694b9e18cc46e7a3736331c8446d71 (127.20.1.188:44741)
I20260620 01:58:31.859098 22458 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:58:31.859329 22458 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:58:31.859709 22458 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:58:31.863778 22458 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } }
I20260620 01:58:31.869074 22398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fb73dffb61a47dca206831b250f5348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1694b9e18cc46e7a3736331c8446d71"
I20260620 01:58:31.869844 22398 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:58:31.876425 22458 leader_election.cc:290] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [CANDIDATE]: Term 1 election: Requested vote from peers fcd356f9391c4f2c806fb32328b70cd1 (127.20.1.189:40165), f1694b9e18cc46e7a3736331c8446d71 (127.20.1.188:44741)
I20260620 01:58:31.877002 22398 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fb73dffb61a47dca206831b250f5348 in term 1.
I20260620 01:58:31.882987 22325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fb73dffb61a47dca206831b250f5348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd356f9391c4f2c806fb32328b70cd1"
I20260620 01:58:31.883430 22325 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:58:31.885979 22230 leader_election.cc:304] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fb73dffb61a47dca206831b250f5348, f1694b9e18cc46e7a3736331c8446d71; no voters:
I20260620 01:58:31.887709 22458 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:58:31.887993 22458 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 1 LEADER]: Becoming Leader. State: Replica: 8fb73dffb61a47dca206831b250f5348, State: Running, Role: LEADER
I20260620 01:58:31.888693 22458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } }
I20260620 01:58:31.889498 22325 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fb73dffb61a47dca206831b250f5348 in term 1.
I20260620 01:58:31.905520 22472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fb73dffb61a47dca206831b250f5348. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } }
I20260620 01:58:31.906095 22472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:31.932169 22474 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:58:31.938985 22474 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:58:31.998670 22398 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 1 FOLLOWER]: Refusing update from remote peer 8fb73dffb61a47dca206831b250f5348: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:58:32.000556 22472 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:58:32.007606 22325 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 1 FOLLOWER]: Refusing update from remote peer 8fb73dffb61a47dca206831b250f5348: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:58:32.009338 22472 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 01:58:32.039333 22443 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fb73dffb61a47dca206831b250f5348. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } }
I20260620 01:58:32.039666 22443 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:32.059607 22476 mvcc.cc:204] Tried to move back new op lower bound from 7298747236285018112 to 7298747235903062016. Current Snapshot: MvccSnapshot[applied={T|T < 7298747236285018112}]
I20260620 01:58:32.064069 22458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } }
I20260620 01:58:32.064409 22458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:32.067152 22474 catalog_manager.cc:1382] Generated new cluster ID: 70aac4e312c24af7a8ab590288cff567
I20260620 01:58:32.067318 22474 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:58:32.091706 22443 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } }
I20260620 01:58:32.092033 22443 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:32.112217 22432 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fb73dffb61a47dca206831b250f5348. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } }
I20260620 01:58:32.112589 22432 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:32.113677 22474 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:58:32.117213 22474 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:58:32.118422 22458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } }
I20260620 01:58:32.128454 22458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:32.228181 22432 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } }
I20260620 01:58:32.228636 22432 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:32.267884 22474 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348: Generated new TSK 0
I20260620 01:58:32.270196 22474 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:58:32.334506 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741
--time_source=system_unsync with env {}
I20260620 01:58:32.747251 22444 catalog_manager.cc:1294] Loaded cluster ID: 70aac4e312c24af7a8ab590288cff567
I20260620 01:58:32.747534 22444 catalog_manager.cc:1587] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1: loading cluster ID for follower catalog manager: success
I20260620 01:58:32.754076 22444 catalog_manager.cc:1609] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1: acquiring CA information for follower catalog manager: success
I20260620 01:58:32.759228 22444 catalog_manager.cc:1637] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:58:32.805716 22457 catalog_manager.cc:1294] Loaded cluster ID: 70aac4e312c24af7a8ab590288cff567
I20260620 01:58:32.805912 22457 catalog_manager.cc:1587] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71: loading cluster ID for follower catalog manager: success
I20260620 01:58:32.813431 22457 catalog_manager.cc:1609] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71: acquiring CA information for follower catalog manager: success
I20260620 01:58:32.818665 22457 catalog_manager.cc:1637] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 01:58:33.021878 22484 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:58:33.022241 22484 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:58:33.022383 22484 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:58:33.033437 22484 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:58:33.033671 22484 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:58:33.049000 22484 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:58:33.052263 22484 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:58:33.055861 22484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:58:33.069078 22490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:33.069028 22489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:33.070019 22492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:58:33.071010 22484 server_base.cc:1061] running on GCE node
I20260620 01:58:33.072199 22484 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:58:33.072427 22484 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:58:33.072536 22484 hybrid_clock.cc:648] HybridClock initialized: now 1781920713072514 us; error 0 us; skew 500 ppm
I20260620 01:58:33.073005 22484 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:58:33.077270 22484 webserver.cc:533] Webserver started at http://127.20.1.129:38821/ using document root <none> and password file <none>
I20260620 01:58:33.078851 22484 fs_manager.cc:362] Metadata directory not provided
I20260620 01:58:33.079025 22484 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:58:33.079551 22484 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:58:33.082665 22484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "b49e0d0b35a745808386912e6e773289"
format_stamp: "Formatted at 2026-06-20 01:58:33 on dist-test-slave-wg14"
I20260620 01:58:33.083760 22484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "b49e0d0b35a745808386912e6e773289"
format_stamp: "Formatted at 2026-06-20 01:58:33 on dist-test-slave-wg14"
I20260620 01:58:33.095695 22484 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.011s sys 0.000s
I20260620 01:58:33.101938 22497 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:33.104419 22484 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:58:33.104665 22484 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "b49e0d0b35a745808386912e6e773289"
format_stamp: "Formatted at 2026-06-20 01:58:33 on dist-test-slave-wg14"
I20260620 01:58:33.104928 22484 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:58:33.163791 22484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:58:33.165002 22484 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:58:33.165342 22484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:58:33.166740 22484 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:58:33.169379 22484 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:58:33.169759 22484 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:33.169965 22484 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:58:33.170042 22484 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:33.228193 22484 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:40469
I20260620 01:58:33.228384 22609 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:40469 every 8 connection(s)
I20260620 01:58:33.230284 22484 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:58:33.236860 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 22484
I20260620 01:58:33.237171 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:58:33.278517 22611 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44173
I20260620 01:58:33.278756 22610 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44741
I20260620 01:58:33.279072 22611 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:33.279153 22610 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:33.280258 22612 heartbeater.cc:344] Connected to a master server at 127.20.1.189:40165
I20260620 01:58:33.280560 22612 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:33.280966 22611 heartbeater.cc:507] Master 127.20.1.190:44173 requested a full tablet report, sending...
I20260620 01:58:33.281042 22610 heartbeater.cc:507] Master 127.20.1.188:44741 requested a full tablet report, sending...
I20260620 01:58:33.281256 22612 heartbeater.cc:507] Master 127.20.1.189:40165 requested a full tablet report, sending...
I20260620 01:58:33.284626 22388 ts_manager.cc:194] Registered new tserver with Master: b49e0d0b35a745808386912e6e773289 (127.20.1.129:40469)
I20260620 01:58:33.284612 22245 ts_manager.cc:194] Registered new tserver with Master: b49e0d0b35a745808386912e6e773289 (127.20.1.129:40469)
I20260620 01:58:33.286902 22315 ts_manager.cc:194] Registered new tserver with Master: b49e0d0b35a745808386912e6e773289 (127.20.1.129:40469)
I20260620 01:58:33.287240 22245 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:51105
I20260620 01:58:33.307518 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:58:33.310325 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:32825
--webserver_interface=127.20.1.187
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741,127.20.1.187:32825 with env {}
W20260620 01:58:33.780798 22621 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:58:33.781196 22621 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:58:33.781276 22621 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:58:33.792078 22621 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:58:33.792227 22621 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:58:33.792290 22621 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:58:33.792332 22621 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:58:33.804517 22621 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741,127.20.1.187:32825
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:32825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.20.1.187
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:58:33.806474 22621 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:58:33.808792 22621 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:58:33.824046 22626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:33.824275 22627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:33.825356 22629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:58:33.827477 22621 server_base.cc:1061] running on GCE node
I20260620 01:58:33.828603 22621 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:58:33.828840 22621 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:58:33.828918 22621 hybrid_clock.cc:648] HybridClock initialized: now 1781920713828896 us; error 0 us; skew 500 ppm
I20260620 01:58:33.829526 22621 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:58:33.833993 22621 webserver.cc:533] Webserver started at http://127.20.1.187:41663/ using document root <none> and password file <none>
I20260620 01:58:33.835050 22621 fs_manager.cc:362] Metadata directory not provided
I20260620 01:58:33.835227 22621 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:58:33.835702 22621 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:58:33.843675 22621 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/data/instance:
uuid: "3d5b20a61b4544f5a62dba46ae1b38d3"
format_stamp: "Formatted at 2026-06-20 01:58:33 on dist-test-slave-wg14"
I20260620 01:58:33.845418 22621 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/wal/instance:
uuid: "3d5b20a61b4544f5a62dba46ae1b38d3"
format_stamp: "Formatted at 2026-06-20 01:58:33 on dist-test-slave-wg14"
I20260620 01:58:33.854791 22621 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260620 01:58:33.867877 22634 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:33.870198 22621 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260620 01:58:33.870626 22621 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "3d5b20a61b4544f5a62dba46ae1b38d3"
format_stamp: "Formatted at 2026-06-20 01:58:33 on dist-test-slave-wg14"
I20260620 01:58:33.871223 22621 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:58:33.899216 22621 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:58:33.900465 22621 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:58:33.900804 22621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:58:33.946851 22621 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:32825
I20260620 01:58:33.946945 22685 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:32825 every 8 connection(s)
I20260620 01:58:33.949699 22621 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 01:58:33.956717 22686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:58:33.960091 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 22621
I20260620 01:58:33.971706 22686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:58:33.998554 22686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:58:34.010775 22686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:58:34.030479 22686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:58:34.048355 22686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: Bootstrap starting.
I20260620 01:58:34.052603 22686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: Neither blocks nor log segments found. Creating new log.
I20260620 01:58:34.054528 22686 log.cc:826] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: Log is configured to *not* fsync() on all Append() calls
I20260620 01:58:34.060125 22686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: No bootstrap required, opened a new log
I20260620 01:58:34.073520 22686 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } }
I20260620 01:58:34.074003 22686 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:58:34.074107 22686 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5b20a61b4544f5a62dba46ae1b38d3, State: Initialized, Role: FOLLOWER
I20260620 01:58:34.074988 22686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } }
I20260620 01:58:34.076709 22695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } } }
I20260620 01:58:34.077219 22695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:34.078215 22686 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:58:34.102874 22706 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:58:34.103525 22706 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:58:34.104210 22695 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:58:34.104548 22695 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } }
I20260620 01:58:34.107568 22695 leader_election.cc:290] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fb73dffb61a47dca206831b250f5348 (127.20.1.190:44173), fcd356f9391c4f2c806fb32328b70cd1 (127.20.1.189:40165), f1694b9e18cc46e7a3736331c8446d71 (127.20.1.188:44741)
I20260620 01:58:34.108440 22397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1694b9e18cc46e7a3736331c8446d71" is_pre_election: true
I20260620 01:58:34.108752 22325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd356f9391c4f2c806fb32328b70cd1" is_pre_election: true
I20260620 01:58:34.109534 22255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fb73dffb61a47dca206831b250f5348" is_pre_election: true
I20260620 01:58:34.110337 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 22484
I20260620 01:58:34.110817 22635 leader_election.cc:304] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3d5b20a61b4544f5a62dba46ae1b38d3; no voters: 8fb73dffb61a47dca206831b250f5348, f1694b9e18cc46e7a3736331c8446d71
I20260620 01:58:34.111573 22695 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 01:58:34.121780 22621 master_runner.cc:429] Detected that this master 3d5b20a61b4544f5a62dba46ae1b38d3 is joining an existing cluster
I20260620 01:58:34.121953 22621 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260620 01:58:34.125831 22621 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 01:58:34.126830 22621 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 01:58:34.127231 22621 tablet_replica.cc:333] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: stopping tablet replica
I20260620 01:58:34.130788 22621 master_runner.cc:434] Initiating AddMaster RPC to add 127.20.1.187:32825
I20260620 01:58:34.134631 22605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:58:34.251797 22245 catalog_manager.cc:7201] Initiating ChangeConfig request to add master 127.20.1.187:32825: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3"
member_type: NON_VOTER
last_known_addr {
host: "127.20.1.187"
port: 32825
}
attrs {
promote: true
}
}
dest_uuid: "8fb73dffb61a47dca206831b250f5348"
cas_config_opid_index: -1
I20260620 01:58:34.253427 22245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: true } }
I20260620 01:58:34.260505 22324 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 1 FOLLOWER]: Refusing update from remote peer 8fb73dffb61a47dca206831b250f5348: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 01:58:34.261790 22620 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260620 01:58:34.262720 22232 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 -> Peer 3d5b20a61b4544f5a62dba46ae1b38d3 (127.20.1.187:32825): Couldn't send request to peer 3d5b20a61b4544f5a62dba46ae1b38d3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260620 01:58:34.265815 22397 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 1 FOLLOWER]: Refusing update from remote peer 8fb73dffb61a47dca206831b250f5348: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 01:58:34.267535 22620 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:58:34.288666 22620 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3d5b20a61b4544f5a62dba46ae1b38d3 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: true } } }
I20260620 01:58:34.290643 22324 raft_consensus.cc:2955] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3d5b20a61b4544f5a62dba46ae1b38d3 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: true } } }
I20260620 01:58:34.295902 22398 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3d5b20a61b4544f5a62dba46ae1b38d3 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: true } } }
I20260620 01:58:34.307439 22620 catalog_manager.cc:7205] Successfully completed master ChangeConfig request to add master 127.20.1.187:32825
I20260620 01:58:34.308923 22724 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: true } } }
I20260620 01:58:34.309427 22621 master.cc:562] Master@127.20.1.187:32825 shutting down...
I20260620 01:58:34.309691 22724 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:34.324764 22721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: true } } }
I20260620 01:58:34.325173 22721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:34.329752 22725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: true } } }
I20260620 01:58:34.330320 22725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:34.332057 22725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: true } } }
I20260620 01:58:34.333184 22725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:34.342950 22611 heartbeater.cc:499] Master 127.20.1.190:44173 was elected leader, sending a full tablet report...
I20260620 01:58:34.344884 22727 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: true } } }
I20260620 01:58:34.345417 22727 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:34.359390 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 22484
I20260620 01:58:34.376065 22621 master.cc:584] Master@127.20.1.187:32825 shutdown complete.
I20260620 01:58:34.376225 22621 master_runner.cc:306] Clearing existing system tablet
I20260620 01:58:34.378434 22621 fs_manager.cc:362] Metadata directory not provided
I20260620 01:58:34.378664 22621 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:58:34.391245 22621 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 01:58:34.393787 22730 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:34.394728 22621 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:58:34.395102 22621 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 01:58:34.395392 22621 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "3d5b20a61b4544f5a62dba46ae1b38d3"
format_stamp: "Formatted at 2026-06-20 01:58:33 on dist-test-slave-wg14"
I20260620 01:58:34.395541 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:40469
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741
--webserver_port=38821
--webserver_interface=127.20.1.129
--time_source=system_unsync
--tserver_master_addrs=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741,127.20.1.187:32825 with env {}
I20260620 01:58:34.408443 22621 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 01:58:34.416584 22621 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 01:58:34.417016 22621 log.cc:1199] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: Deleting WAL directory at /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260620 01:58:34.417876 22621 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: Deleting consensus metadata
I20260620 01:58:34.418689 22621 master_runner.cc:316] Copying system tablet from 127.20.1.190:44173
I20260620 01:58:34.429949 22621 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.1.190:44173
I20260620 01:58:34.448858 22265 tablet_copy_service.cc:140] P 8fb73dffb61a47dca206831b250f5348: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3d5b20a61b4544f5a62dba46ae1b38d3 ({username='slave'} at 127.0.0.1:38238)
I20260620 01:58:34.449169 22265 tablet_copy_service.cc:161] P 8fb73dffb61a47dca206831b250f5348: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3d5b20a61b4544f5a62dba46ae1b38d3 at {username='slave'} at 127.0.0.1:38238: session id = 3d5b20a61b4544f5a62dba46ae1b38d3-00000000000000000000000000000000
I20260620 01:58:34.452642 22265 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 01:58:34.455471 22621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 01:58:34.469761 22621 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: tablet copy: Starting download of 0 data blocks...
I20260620 01:58:34.470132 22621 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: tablet copy: Starting download of 1 WAL segments...
I20260620 01:58:34.473872 22621 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 01:58:34.481004 22265 tablet_copy_service.cc:342] P 8fb73dffb61a47dca206831b250f5348: Request end of tablet copy session 3d5b20a61b4544f5a62dba46ae1b38d3-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:38238
I20260620 01:58:34.481696 22265 tablet_copy_service.cc:434] P 8fb73dffb61a47dca206831b250f5348: ending tablet copy session 3d5b20a61b4544f5a62dba46ae1b38d3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3d5b20a61b4544f5a62dba46ae1b38d3
I20260620 01:58:34.495584 22621 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:58:34.499033 22738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:34.505734 22739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:34.525444 22741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:58:34.525723 22621 server_base.cc:1061] running on GCE node
I20260620 01:58:34.526036 22621 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:58:34.526129 22621 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:58:34.526192 22621 hybrid_clock.cc:648] HybridClock initialized: now 1781920714526166 us; error 0 us; skew 500 ppm
I20260620 01:58:34.526393 22621 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:58:34.528314 22621 webserver.cc:533] Webserver started at http://127.20.1.187:38399/ using document root <none> and password file <none>
I20260620 01:58:34.528728 22621 fs_manager.cc:362] Metadata directory not provided
I20260620 01:58:34.528812 22621 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:58:34.531585 22621 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 01:58:34.533668 22746 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:34.534287 22621 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 01:58:34.534474 22621 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "3d5b20a61b4544f5a62dba46ae1b38d3"
format_stamp: "Formatted at 2026-06-20 01:58:33 on dist-test-slave-wg14"
I20260620 01:58:34.534875 22621 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:58:34.555037 22621 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:58:34.556129 22621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:58:34.620170 22621 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:32825
I20260620 01:58:34.620626 22797 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:32825 every 8 connection(s)
I20260620 01:58:34.621599 22621 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 01:58:34.627830 22798 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:58:34.634277 22798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: Bootstrap starting.
I20260620 01:58:34.664076 22798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:58:34.664908 22798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: Bootstrap complete.
I20260620 01:58:34.666651 22798 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: true } }
I20260620 01:58:34.667798 22798 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d5b20a61b4544f5a62dba46ae1b38d3, State: Initialized, Role: LEARNER
I20260620 01:58:34.668852 22798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: true } }
I20260620 01:58:34.672013 22801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: true } } }
I20260620 01:58:34.672609 22801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [sys.catalog]: This master's current role is: LEARNER
I20260620 01:58:34.672971 22798 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:58:34.698227 22812 catalog_manager.cc:1294] Loaded cluster ID: 70aac4e312c24af7a8ab590288cff567
I20260620 01:58:34.698589 22812 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: loading cluster ID for follower catalog manager: success
I20260620 01:58:34.702390 22812 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: acquiring CA information for follower catalog manager: success
I20260620 01:58:34.705610 22812 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:58:34.724228 22773 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260620 01:58:34.725098 22801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fb73dffb61a47dca206831b250f5348. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: true } } }
I20260620 01:58:34.725441 22801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [sys.catalog]: This master's current role is: LEARNER
W20260620 01:58:34.989990 22731 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:58:34.990428 22731 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:58:34.990559 22731 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:58:35.002235 22731 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:58:35.002477 22731 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:58:35.017401 22731 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:40469
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=38821
--tserver_master_addrs=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741,127.20.1.187:32825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:58:35.019618 22731 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:58:35.022119 22731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:58:35.037468 22818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:35.037530 22819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:35.038434 22821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:58:35.039582 22731 server_base.cc:1061] running on GCE node
I20260620 01:58:35.040491 22731 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:58:35.040721 22731 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:58:35.040843 22731 hybrid_clock.cc:648] HybridClock initialized: now 1781920715040813 us; error 0 us; skew 500 ppm
I20260620 01:58:35.041215 22731 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:58:35.044620 22731 webserver.cc:533] Webserver started at http://127.20.1.129:38821/ using document root <none> and password file <none>
I20260620 01:58:35.045753 22731 fs_manager.cc:362] Metadata directory not provided
I20260620 01:58:35.045950 22731 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:58:35.054265 22731 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 01:58:35.059348 22826 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:35.061568 22731 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:58:35.061882 22731 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "b49e0d0b35a745808386912e6e773289"
format_stamp: "Formatted at 2026-06-20 01:58:33 on dist-test-slave-wg14"
I20260620 01:58:35.062769 22731 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:58:35.092305 22731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:58:35.093807 22731 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:58:35.094385 22731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:58:35.095898 22731 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:58:35.098327 22731 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:58:35.098452 22731 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:35.098588 22731 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:58:35.098639 22731 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:35.151839 22731 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:40469
I20260620 01:58:35.151918 22938 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:40469 every 8 connection(s)
I20260620 01:58:35.154929 22731 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:58:35.162436 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 22731
I20260620 01:58:35.206243 22940 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44741
I20260620 01:58:35.206888 22940 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:35.209794 22940 heartbeater.cc:507] Master 127.20.1.188:44741 requested a full tablet report, sending...
I20260620 01:58:35.213220 22953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: true } } }
I20260620 01:58:35.213569 22388 ts_manager.cc:194] Re-registered known tserver with Master: b49e0d0b35a745808386912e6e773289 (127.20.1.129:40469)
I20260620 01:58:35.213860 22953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:35.215237 22942 heartbeater.cc:344] Connected to a master server at 127.20.1.189:40165
I20260620 01:58:35.215512 22942 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:35.216395 22942 heartbeater.cc:507] Master 127.20.1.189:40165 requested a full tablet report, sending...
I20260620 01:58:35.219233 22953 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348: attempting to promote NON_VOTER 3d5b20a61b4544f5a62dba46ae1b38d3 to VOTER
I20260620 01:58:35.220450 22315 ts_manager.cc:194] Re-registered known tserver with Master: b49e0d0b35a745808386912e6e773289 (127.20.1.129:40469)
I20260620 01:58:35.220541 22941 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44173
I20260620 01:58:35.220717 22941 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:35.221477 22941 heartbeater.cc:507] Master 127.20.1.190:44173 requested a full tablet report, sending...
I20260620 01:58:35.222065 22953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } }
I20260620 01:58:35.222556 22939 heartbeater.cc:344] Connected to a master server at 127.20.1.187:32825
I20260620 01:58:35.222741 22939 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:35.223466 22939 heartbeater.cc:507] Master 127.20.1.187:32825 requested a full tablet report, sending...
I20260620 01:58:35.225107 22245 ts_manager.cc:194] Re-registered known tserver with Master: b49e0d0b35a745808386912e6e773289 (127.20.1.129:40469)
I20260620 01:58:35.225989 22762 ts_manager.cc:194] Registered new tserver with Master: b49e0d0b35a745808386912e6e773289 (127.20.1.129:40469)
I20260620 01:58:35.226049 22398 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 1 FOLLOWER]: Refusing update from remote peer 8fb73dffb61a47dca206831b250f5348: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 01:58:35.226688 22245 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:35417
I20260620 01:58:35.228142 22957 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 01:58:35.229619 22773 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 1 LEARNER]: Refusing update from remote peer 8fb73dffb61a47dca206831b250f5348: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 01:58:35.229705 22324 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 1 FOLLOWER]: Refusing update from remote peer 8fb73dffb61a47dca206831b250f5348: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 01:58:35.230859 22957 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:58:35.231333 22956 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:58:35.236022 22957 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3d5b20a61b4544f5a62dba46ae1b38d3 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:35.236785 22324 raft_consensus.cc:2955] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3d5b20a61b4544f5a62dba46ae1b38d3 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:35.238085 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:58:35.239095 22773 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3d5b20a61b4544f5a62dba46ae1b38d3 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:35.241847 22957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:35.243919 22957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:35.243490 22398 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3d5b20a61b4544f5a62dba46ae1b38d3 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:35.243670 22953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:35.244294 22953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:35.243096 22954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:35.244514 22954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:35.244939 22953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:35.245371 22953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:35.246173 22959 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:35.246505 22959 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:35.247087 22960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:35.247521 22960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:35.249063 22961 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fb73dffb61a47dca206831b250f5348" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:35.249423 22961 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:35.257350 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:58:35.257568 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 22215
I20260620 01:58:35.258891 22275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:58:35.403347 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 22215
I20260620 01:58:35.427004 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44173
--webserver_interface=127.20.1.190
--webserver_port=42499
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741
--master_addresses=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741,127.20.1.187:32825 with env {}
W20260620 01:58:35.951483 22963 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:58:35.951922 22963 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:58:35.952068 22963 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:58:35.962706 22963 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:58:35.962883 22963 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:58:35.962950 22963 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:58:35.963018 22963 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:58:35.978029 22963 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741,127.20.1.187:32825
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=42499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:58:35.979948 22963 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:58:35.982614 22963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:58:36.015486 22969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:36.016113 22971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:36.015872 22968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:58:36.025816 22963 server_base.cc:1061] running on GCE node
I20260620 01:58:36.027694 22963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:58:36.027972 22963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:58:36.028088 22963 hybrid_clock.cc:648] HybridClock initialized: now 1781920716028060 us; error 0 us; skew 500 ppm
I20260620 01:58:36.028532 22963 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:58:36.032827 22963 webserver.cc:533] Webserver started at http://127.20.1.190:42499/ using document root <none> and password file <none>
I20260620 01:58:36.034040 22963 fs_manager.cc:362] Metadata directory not provided
I20260620 01:58:36.034246 22963 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:58:36.048250 22963 fs_manager.cc:714] Time spent opening directory manager: real 0.009s user 0.008s sys 0.000s
I20260620 01:58:36.052901 22976 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:36.055529 22963 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 01:58:36.055811 22963 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "8fb73dffb61a47dca206831b250f5348"
format_stamp: "Formatted at 2026-06-20 01:58:29 on dist-test-slave-wg14"
I20260620 01:58:36.056852 22963 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:58:36.120054 22963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:58:36.124320 22963 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:58:36.125041 22963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:58:36.190021 23027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44173 every 8 connection(s)
I20260620 01:58:36.190136 22963 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44173
I20260620 01:58:36.192785 22963 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:58:36.197873 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 22963
I20260620 01:58:36.198551 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 22731
I20260620 01:58:36.204562 23028 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:58:36.210891 23028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348: Bootstrap starting.
I20260620 01:58:36.218973 22934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:58:36.231302 22941 heartbeater.cc:499] Master 127.20.1.190:44173 was elected leader, sending a full tablet report...
I20260620 01:58:36.244668 23028 log.cc:826] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348: Log is configured to *not* fsync() on all Append() calls
I20260620 01:58:36.269299 23028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:58:36.270731 23028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348: Bootstrap complete.
I20260620 01:58:36.286218 23028 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } }
I20260620 01:58:36.287928 23028 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fb73dffb61a47dca206831b250f5348, State: Initialized, Role: FOLLOWER
I20260620 01:58:36.290246 23028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } }
I20260620 01:58:36.291646 23034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:36.292097 23034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:36.296439 23028 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:58:36.320951 23045 catalog_manager.cc:1294] Loaded cluster ID: 70aac4e312c24af7a8ab590288cff567
I20260620 01:58:36.321142 23045 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348: loading cluster ID for follower catalog manager: success
I20260620 01:58:36.328800 23045 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348: acquiring CA information for follower catalog manager: success
I20260620 01:58:36.333997 23045 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:58:36.405797 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 22731
I20260620 01:58:36.439061 22963 master_runner.cc:264] No leader master found from master 8fb73dffb61a47dca206831b250f5348
I20260620 01:58:36.439289 22963 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:58:36.449352 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:40469
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741
--webserver_port=38821
--webserver_interface=127.20.1.129
--time_source=system_unsync
--tserver_master_addrs=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741,127.20.1.187:32825 with env {}
W20260620 01:58:36.449765 23048 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:58:36.399386 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:55469 (local address 127.20.1.190:44173)
0620 01:58:36.399587 (+ 201us) server_negotiation.cc:207] Beginning negotiation
0620 01:58:36.399593 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0620 01:58:36.405199 (+ 5606us) server_negotiation.cc:408] Connection header received
0620 01:58:36.447824 (+ 42625us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:55469: BlockingRecv error: recv got EOF from 127.20.1.129:55469 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 01:58:36.751852 23055 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8fb73dffb61a47dca206831b250f5348)
I20260620 01:58:36.754313 23055 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } }
I20260620 01:58:36.780510 23055 leader_election.cc:290] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8fb73dffb61a47dca206831b250f5348 (127.20.1.190:44173), fcd356f9391c4f2c806fb32328b70cd1 (127.20.1.189:40165), f1694b9e18cc46e7a3736331c8446d71 (127.20.1.188:44741)
I20260620 01:58:36.781848 23057 raft_consensus.cc:493] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8fb73dffb61a47dca206831b250f5348)
I20260620 01:58:36.782150 23057 raft_consensus.cc:515] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } }
I20260620 01:58:36.786713 23054 raft_consensus.cc:493] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8fb73dffb61a47dca206831b250f5348)
I20260620 01:58:36.811357 22324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "fcd356f9391c4f2c806fb32328b70cd1" is_pre_election: true
I20260620 01:58:36.811594 22398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "f1694b9e18cc46e7a3736331c8446d71" is_pre_election: true
I20260620 01:58:36.811874 22398 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d5b20a61b4544f5a62dba46ae1b38d3 in term 1.
I20260620 01:58:36.811396 22325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1694b9e18cc46e7a3736331c8446d71" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "fcd356f9391c4f2c806fb32328b70cd1" is_pre_election: true
I20260620 01:58:36.812337 23054 raft_consensus.cc:515] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } }
I20260620 01:58:36.813185 22324 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d5b20a61b4544f5a62dba46ae1b38d3 in term 1.
I20260620 01:58:36.817410 23057 leader_election.cc:290] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8fb73dffb61a47dca206831b250f5348 (127.20.1.190:44173), fcd356f9391c4f2c806fb32328b70cd1 (127.20.1.189:40165), 3d5b20a61b4544f5a62dba46ae1b38d3 (127.20.1.187:32825)
I20260620 01:58:36.817767 23003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcd356f9391c4f2c806fb32328b70cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "8fb73dffb61a47dca206831b250f5348" is_pre_election: true
I20260620 01:58:36.818360 22398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcd356f9391c4f2c806fb32328b70cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "f1694b9e18cc46e7a3736331c8446d71" is_pre_election: true
I20260620 01:58:36.818542 23003 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcd356f9391c4f2c806fb32328b70cd1 in term 1.
I20260620 01:58:36.818640 22398 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcd356f9391c4f2c806fb32328b70cd1 in term 1.
I20260620 01:58:36.819645 22300 leader_election.cc:304] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8fb73dffb61a47dca206831b250f5348, f1694b9e18cc46e7a3736331c8446d71, fcd356f9391c4f2c806fb32328b70cd1; no voters:
I20260620 01:58:36.833704 23054 leader_election.cc:290] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8fb73dffb61a47dca206831b250f5348 (127.20.1.190:44173), f1694b9e18cc46e7a3736331c8446d71 (127.20.1.188:44741), 3d5b20a61b4544f5a62dba46ae1b38d3 (127.20.1.187:32825)
I20260620 01:58:36.835609 23054 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:58:36.837626 23054 raft_consensus.cc:493] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8fb73dffb61a47dca206831b250f5348)
I20260620 01:58:36.838297 23054 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:58:36.844498 22748 leader_election.cc:304] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d5b20a61b4544f5a62dba46ae1b38d3, f1694b9e18cc46e7a3736331c8446d71, fcd356f9391c4f2c806fb32328b70cd1; no voters:
I20260620 01:58:36.845053 23055 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 01:58:36.845208 23055 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8fb73dffb61a47dca206831b250f5348)
I20260620 01:58:36.845355 23055 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:58:36.846220 23003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1694b9e18cc46e7a3736331c8446d71" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "8fb73dffb61a47dca206831b250f5348" is_pre_election: true
I20260620 01:58:36.846516 23003 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1694b9e18cc46e7a3736331c8446d71 in term 1.
I20260620 01:58:36.852582 23055 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } }
I20260620 01:58:36.843083 23054 raft_consensus.cc:515] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } }
I20260620 01:58:36.858712 22324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "fcd356f9391c4f2c806fb32328b70cd1"
I20260620 01:58:36.859110 22324 raft_consensus.cc:2393] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d5b20a61b4544f5a62dba46ae1b38d3 in current term 2: Already voted for candidate fcd356f9391c4f2c806fb32328b70cd1 in this term.
I20260620 01:58:36.859586 23055 leader_election.cc:290] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [CANDIDATE]: Term 2 election: Requested vote from peers 8fb73dffb61a47dca206831b250f5348 (127.20.1.190:44173), fcd356f9391c4f2c806fb32328b70cd1 (127.20.1.189:40165), f1694b9e18cc46e7a3736331c8446d71 (127.20.1.188:44741)
I20260620 01:58:36.860898 22398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "f1694b9e18cc46e7a3736331c8446d71"
I20260620 01:58:36.861228 22398 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:58:36.862586 23003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcd356f9391c4f2c806fb32328b70cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "8fb73dffb61a47dca206831b250f5348"
I20260620 01:58:36.862900 23003 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 1 FOLLOWER]: Advancing to term 2
I20260620 01:58:36.863077 22397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcd356f9391c4f2c806fb32328b70cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "f1694b9e18cc46e7a3736331c8446d71"
I20260620 01:58:36.864893 22398 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d5b20a61b4544f5a62dba46ae1b38d3 in term 2.
I20260620 01:58:36.865819 23054 leader_election.cc:290] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 8fb73dffb61a47dca206831b250f5348 (127.20.1.190:44173), f1694b9e18cc46e7a3736331c8446d71 (127.20.1.188:44741), 3d5b20a61b4544f5a62dba46ae1b38d3 (127.20.1.187:32825)
I20260620 01:58:36.869884 22352 master_runner.cc:233] Existing masters have differing terms: 1 vs 2
I20260620 01:58:36.870244 22352 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:58:36.871429 22282 master_runner.cc:233] Existing masters have differing terms: 1 vs 2
I20260620 01:58:36.871745 22282 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:58:36.872543 23003 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcd356f9391c4f2c806fb32328b70cd1 in term 2.
I20260620 01:58:36.884341 23002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "8fb73dffb61a47dca206831b250f5348"
I20260620 01:58:36.884949 23002 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d5b20a61b4544f5a62dba46ae1b38d3 in current term 2: Already voted for candidate fcd356f9391c4f2c806fb32328b70cd1 in this term.
I20260620 01:58:36.886087 22747 leader_election.cc:304] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3d5b20a61b4544f5a62dba46ae1b38d3, f1694b9e18cc46e7a3736331c8446d71; no voters: 8fb73dffb61a47dca206831b250f5348, fcd356f9391c4f2c806fb32328b70cd1
I20260620 01:58:36.886824 23055 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 01:58:36.887945 23003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "8fb73dffb61a47dca206831b250f5348" is_pre_election: true
I20260620 01:58:36.888339 23003 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d5b20a61b4544f5a62dba46ae1b38d3 in current term 2: Already voted for candidate fcd356f9391c4f2c806fb32328b70cd1 in this term.
I20260620 01:58:36.888783 22773 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcd356f9391c4f2c806fb32328b70cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" is_pre_election: true
I20260620 01:58:36.889209 22773 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fcd356f9391c4f2c806fb32328b70cd1 in current term 2: Already voted for candidate 3d5b20a61b4544f5a62dba46ae1b38d3 in this term.
I20260620 01:58:36.889950 22773 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcd356f9391c4f2c806fb32328b70cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3"
I20260620 01:58:36.890225 22773 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fcd356f9391c4f2c806fb32328b70cd1 in current term 2: Already voted for candidate 3d5b20a61b4544f5a62dba46ae1b38d3 in this term.
I20260620 01:58:36.890995 22773 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1694b9e18cc46e7a3736331c8446d71" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" is_pre_election: true
I20260620 01:58:36.891332 22773 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1694b9e18cc46e7a3736331c8446d71 in current term 2: Already voted for candidate 3d5b20a61b4544f5a62dba46ae1b38d3 in this term.
I20260620 01:58:36.891853 22375 leader_election.cc:304] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8fb73dffb61a47dca206831b250f5348, f1694b9e18cc46e7a3736331c8446d71; no voters: 3d5b20a61b4544f5a62dba46ae1b38d3, fcd356f9391c4f2c806fb32328b70cd1
I20260620 01:58:36.892351 22302 leader_election.cc:304] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8fb73dffb61a47dca206831b250f5348, fcd356f9391c4f2c806fb32328b70cd1; no voters: 3d5b20a61b4544f5a62dba46ae1b38d3, f1694b9e18cc46e7a3736331c8446d71
I20260620 01:58:36.892652 23057 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 01:58:36.893137 23054 raft_consensus.cc:2749] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260620 01:58:37.069336 23053 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:58:37.069744 23053 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:58:37.069864 23053 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:58:37.081230 23053 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:58:37.081502 23053 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:58:37.095249 23053 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:40469
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=38821
--tserver_master_addrs=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741,127.20.1.187:32825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:58:37.097565 23053 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:58:37.100220 23053 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:58:37.117559 23072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:37.117690 23074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:37.118170 23071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:58:37.118427 23053 server_base.cc:1061] running on GCE node
I20260620 01:58:37.119377 23053 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:58:37.119585 23053 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:58:37.119700 23053 hybrid_clock.cc:648] HybridClock initialized: now 1781920717119674 us; error 0 us; skew 500 ppm
I20260620 01:58:37.120098 23053 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:58:37.123409 23053 webserver.cc:533] Webserver started at http://127.20.1.129:38821/ using document root <none> and password file <none>
I20260620 01:58:37.124491 23053 fs_manager.cc:362] Metadata directory not provided
I20260620 01:58:37.124655 23053 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:58:37.133419 23053 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 01:58:37.138314 23079 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:37.141145 23053 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:58:37.141443 23053 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "b49e0d0b35a745808386912e6e773289"
format_stamp: "Formatted at 2026-06-20 01:58:33 on dist-test-slave-wg14"
I20260620 01:58:37.142720 23053 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:58:37.174691 23053 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:58:37.175882 23053 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:58:37.176222 23053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:58:37.177681 23053 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:58:37.180130 23053 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:58:37.180312 23053 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:37.180434 23053 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:58:37.180492 23053 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:37.253000 23053 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:40469
I20260620 01:58:37.253548 23191 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:40469 every 8 connection(s)
I20260620 01:58:37.256237 23053 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:58:37.265146 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 23053
I20260620 01:58:37.291703 23192 heartbeater.cc:344] Connected to a master server at 127.20.1.187:32825
I20260620 01:58:37.292253 23192 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:37.293818 23192 heartbeater.cc:507] Master 127.20.1.187:32825 requested a full tablet report, sending...
I20260620 01:58:37.296741 22762 ts_manager.cc:194] Re-registered known tserver with Master: b49e0d0b35a745808386912e6e773289 (127.20.1.129:40469)
I20260620 01:58:37.298256 23193 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44741
I20260620 01:58:37.298660 23193 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:37.298835 23195 heartbeater.cc:344] Connected to a master server at 127.20.1.189:40165
I20260620 01:58:37.298967 23195 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:37.299568 23195 heartbeater.cc:507] Master 127.20.1.189:40165 requested a full tablet report, sending...
I20260620 01:58:37.300179 23193 heartbeater.cc:507] Master 127.20.1.188:44741 requested a full tablet report, sending...
I20260620 01:58:37.300560 23194 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44173
I20260620 01:58:37.300722 23194 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:37.301324 23194 heartbeater.cc:507] Master 127.20.1.190:44173 requested a full tablet report, sending...
I20260620 01:58:37.301985 22388 ts_manager.cc:194] Re-registered known tserver with Master: b49e0d0b35a745808386912e6e773289 (127.20.1.129:40469)
I20260620 01:58:37.302831 22315 ts_manager.cc:194] Re-registered known tserver with Master: b49e0d0b35a745808386912e6e773289 (127.20.1.129:40469)
I20260620 01:58:37.304279 22993 ts_manager.cc:194] Registered new tserver with Master: b49e0d0b35a745808386912e6e773289 (127.20.1.129:40469)
I20260620 01:58:37.346096 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 23053
I20260620 01:58:37.364684 23187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:58:37.535682 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 23053
I20260620 01:58:37.565330 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:40469
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741
--webserver_port=38821
--webserver_interface=127.20.1.129
--time_source=system_unsync
--tserver_master_addrs=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741,127.20.1.187:32825
--fault_before_start=1.0 with env {}
W20260620 01:58:38.249236 23205 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:58:38.249720 23205 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:58:38.249876 23205 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:58:38.249972 23205 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:58:38.264652 23205 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:58:38.265264 23205 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:58:38.281100 23205 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:40469
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=38821
--tserver_master_addrs=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741,127.20.1.187:32825
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:58:38.283737 23205 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:58:38.286276 23205 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:58:38.304363 23211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:38.304363 23210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:38.307749 23213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:58:38.310665 23205 server_base.cc:1061] running on GCE node
I20260620 01:58:38.311686 23205 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:58:38.311908 23205 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:58:38.312049 23205 hybrid_clock.cc:648] HybridClock initialized: now 1781920718311996 us; error 0 us; skew 500 ppm
I20260620 01:58:38.312808 23205 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:58:38.316833 23205 webserver.cc:533] Webserver started at http://127.20.1.129:38821/ using document root <none> and password file <none>
I20260620 01:58:38.318476 23205 fs_manager.cc:362] Metadata directory not provided
I20260620 01:58:38.318755 23205 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:58:38.328037 23205 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 01:58:38.332739 23218 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:38.334971 23205 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:58:38.335286 23205 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "b49e0d0b35a745808386912e6e773289"
format_stamp: "Formatted at 2026-06-20 01:58:33 on dist-test-slave-wg14"
I20260620 01:58:38.336588 23205 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:58:38.385787 23205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:58:38.387087 23205 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:58:38.387493 23205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:58:38.388993 23205 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:58:38.391738 23205 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:58:38.391912 23205 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:38.392077 23205 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:58:38.392159 23205 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:58:38.392735 23205 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:58:38.392858 23205 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:58:38.392946 23205 fault_injection.cc:59] injecting fault for kudu (pid 23205): FLAGS_fault_before_start (process will exit)
I20260620 01:58:38.411087 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:40469
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741
--webserver_port=38821
--webserver_interface=127.20.1.129
--time_source=system_unsync
--tserver_master_addrs=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741,127.20.1.187:32825 with env {}
I20260620 01:58:38.554399 23227 raft_consensus.cc:493] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:58:38.554662 23227 raft_consensus.cc:515] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } }
I20260620 01:58:38.555796 23227 leader_election.cc:290] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fb73dffb61a47dca206831b250f5348 (127.20.1.190:44173), fcd356f9391c4f2c806fb32328b70cd1 (127.20.1.189:40165), 3d5b20a61b4544f5a62dba46ae1b38d3 (127.20.1.187:32825)
I20260620 01:58:38.556442 23003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1694b9e18cc46e7a3736331c8446d71" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "8fb73dffb61a47dca206831b250f5348" is_pre_election: true
I20260620 01:58:38.556707 22773 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1694b9e18cc46e7a3736331c8446d71" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" is_pre_election: true
I20260620 01:58:38.556818 23003 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1694b9e18cc46e7a3736331c8446d71 in term 2.
I20260620 01:58:38.557029 22773 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1694b9e18cc46e7a3736331c8446d71 in term 2.
I20260620 01:58:38.557323 22324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1694b9e18cc46e7a3736331c8446d71" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "fcd356f9391c4f2c806fb32328b70cd1" is_pre_election: true
I20260620 01:58:38.557632 22324 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1694b9e18cc46e7a3736331c8446d71 in term 2.
I20260620 01:58:38.557808 22372 leader_election.cc:304] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d5b20a61b4544f5a62dba46ae1b38d3, 8fb73dffb61a47dca206831b250f5348, f1694b9e18cc46e7a3736331c8446d71; no voters:
I20260620 01:58:38.558579 23227 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 01:58:38.558772 23227 raft_consensus.cc:493] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:58:38.558889 23227 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:58:38.564185 23227 raft_consensus.cc:515] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } }
I20260620 01:58:38.565872 23227 leader_election.cc:290] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [CANDIDATE]: Term 3 election: Requested vote from peers 8fb73dffb61a47dca206831b250f5348 (127.20.1.190:44173), fcd356f9391c4f2c806fb32328b70cd1 (127.20.1.189:40165), 3d5b20a61b4544f5a62dba46ae1b38d3 (127.20.1.187:32825)
I20260620 01:58:38.566329 22324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1694b9e18cc46e7a3736331c8446d71" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "fcd356f9391c4f2c806fb32328b70cd1"
I20260620 01:58:38.566634 22324 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:58:38.567436 23003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1694b9e18cc46e7a3736331c8446d71" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "8fb73dffb61a47dca206831b250f5348"
I20260620 01:58:38.567737 23003 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:58:38.568838 22773 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1694b9e18cc46e7a3736331c8446d71" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3"
I20260620 01:58:38.569247 22773 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 2 FOLLOWER]: Advancing to term 3
I20260620 01:58:38.573369 22324 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1694b9e18cc46e7a3736331c8446d71 in term 3.
I20260620 01:58:38.573395 22773 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1694b9e18cc46e7a3736331c8446d71 in term 3.
I20260620 01:58:38.574168 23003 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1694b9e18cc46e7a3736331c8446d71 in term 3.
I20260620 01:58:38.574386 22375 leader_election.cc:304] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d5b20a61b4544f5a62dba46ae1b38d3, f1694b9e18cc46e7a3736331c8446d71, fcd356f9391c4f2c806fb32328b70cd1; no voters:
I20260620 01:58:38.574903 23227 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 01:58:38.575507 23227 raft_consensus.cc:697] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [term 3 LEADER]: Becoming Leader. State: Replica: f1694b9e18cc46e7a3736331c8446d71, State: Running, Role: LEADER
I20260620 01:58:38.576089 23227 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } }
I20260620 01:58:38.580193 23228 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1694b9e18cc46e7a3736331c8446d71. Latest consensus state: current_term: 3 leader_uuid: "f1694b9e18cc46e7a3736331c8446d71" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:38.581525 23228 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:38.584590 23230 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:58:38.587890 23230 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:58:38.589215 23230 catalog_manager.cc:1294] Loaded cluster ID: 70aac4e312c24af7a8ab590288cff567
I20260620 01:58:38.589371 23230 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:58:38.591307 23230 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:58:38.592500 23230 catalog_manager.cc:6080] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71: Loaded TSK: 0
I20260620 01:58:38.594113 23230 catalog_manager.cc:1549] Initializing in-progress tserver states...
W20260620 01:58:38.916797 23226 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:58:38.917202 23226 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:58:38.917289 23226 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:58:38.927074 23226 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:58:38.927328 23226 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:58:38.940304 23226 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:40469
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=38821
--tserver_master_addrs=127.20.1.190:44173,127.20.1.189:40165,127.20.1.188:44741,127.20.1.187:32825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:58:38.942287 23226 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:58:38.944738 23226 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:58:38.967717 23236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:38.967549 23235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:58:38.968662 23238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:58:38.971743 23226 server_base.cc:1061] running on GCE node
I20260620 01:58:38.972977 23226 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:58:38.973201 23226 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:58:38.973320 23226 hybrid_clock.cc:648] HybridClock initialized: now 1781920718973288 us; error 0 us; skew 500 ppm
I20260620 01:58:38.973693 23226 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:58:38.977922 23226 webserver.cc:533] Webserver started at http://127.20.1.129:38821/ using document root <none> and password file <none>
I20260620 01:58:38.979826 23226 fs_manager.cc:362] Metadata directory not provided
I20260620 01:58:38.980145 23226 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:58:38.989677 23226 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.003s sys 0.001s
I20260620 01:58:38.995251 23243 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:38.998238 23226 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260620 01:58:38.998543 23226 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "b49e0d0b35a745808386912e6e773289"
format_stamp: "Formatted at 2026-06-20 01:58:33 on dist-test-slave-wg14"
I20260620 01:58:38.999655 23226 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:58:39.014763 22773 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [term 3 FOLLOWER]: Refusing update from remote peer f1694b9e18cc46e7a3736331c8446d71: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20260620 01:58:39.016451 23228 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 01:58:39.018549 23226 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:58:39.020249 23226 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:58:39.020746 23226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:58:39.027206 23226 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:58:39.027731 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1694b9e18cc46e7a3736331c8446d71. Latest consensus state: current_term: 3 leader_uuid: "f1694b9e18cc46e7a3736331c8446d71" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:39.028141 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:39.030264 23226 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:58:39.030532 23226 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:39.030694 23226 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:58:39.030759 23226 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:58:39.085246 23226 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:40469
I20260620 01:58:39.085348 23357 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:40469 every 8 connection(s)
I20260620 01:58:39.087523 23226 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:58:39.095878 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 23226
I20260620 01:58:39.096243 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 23226
I20260620 01:58:39.123070 22324 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [term 3 FOLLOWER]: Refusing update from remote peer f1694b9e18cc46e7a3736331c8446d71: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20260620 01:58:39.125718 23228 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 01:58:39.138813 23360 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44173
I20260620 01:58:39.139689 23360 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:39.141515 23360 heartbeater.cc:507] Master 127.20.1.190:44173 requested a full tablet report, sending...
I20260620 01:58:39.145375 22993 ts_manager.cc:194] Re-registered known tserver with Master: b49e0d0b35a745808386912e6e773289 (127.20.1.129:40469)
I20260620 01:58:39.151501 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1694b9e18cc46e7a3736331c8446d71. Latest consensus state: current_term: 3 leader_uuid: "f1694b9e18cc46e7a3736331c8446d71" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:39.151939 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:39.155107 23359 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44741
I20260620 01:58:39.155532 23359 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:39.156260 23359 heartbeater.cc:507] Master 127.20.1.188:44741 requested a full tablet report, sending...
I20260620 01:58:39.155236 23358 heartbeater.cc:344] Connected to a master server at 127.20.1.187:32825
I20260620 01:58:39.157083 23358 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:39.157714 23361 heartbeater.cc:344] Connected to a master server at 127.20.1.189:40165
I20260620 01:58:39.158141 23361 heartbeater.cc:461] Registering TS with master...
I20260620 01:58:39.159083 23361 heartbeater.cc:507] Master 127.20.1.189:40165 requested a full tablet report, sending...
I20260620 01:58:39.160414 23358 heartbeater.cc:507] Master 127.20.1.187:32825 requested a full tablet report, sending...
I20260620 01:58:39.162009 22762 ts_manager.cc:194] Re-registered known tserver with Master: b49e0d0b35a745808386912e6e773289 (127.20.1.129:40469)
I20260620 01:58:39.162889 22315 ts_manager.cc:194] Re-registered known tserver with Master: b49e0d0b35a745808386912e6e773289 (127.20.1.129:40469)
I20260620 01:58:39.163401 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f1694b9e18cc46e7a3736331c8446d71" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:39.163671 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd356f9391c4f2c806fb32328b70cd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:39.164752 23003 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [term 3 FOLLOWER]: Refusing update from remote peer f1694b9e18cc46e7a3736331c8446d71: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20260620 01:58:39.166131 23375 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 01:58:39.167856 22388 ts_manager.cc:194] Re-registered known tserver with Master: b49e0d0b35a745808386912e6e773289 (127.20.1.129:40469)
I20260620 01:58:39.168170 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f1694b9e18cc46e7a3736331c8446d71" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:39.168416 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5b20a61b4544f5a62dba46ae1b38d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:39.170186 23353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:58:39.170559 23228 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f1694b9e18cc46e7a3736331c8446d71" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:39.170644 23375 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f1694b9e18cc46e7a3736331c8446d71" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:39.171231 23228 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:39.171424 23375 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: This master's current role is: LEADER
I20260620 01:58:39.171720 22388 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:54373
I20260620 01:58:39.177642 23382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1694b9e18cc46e7a3736331c8446d71. Latest consensus state: current_term: 3 leader_uuid: "f1694b9e18cc46e7a3736331c8446d71" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:39.177994 23382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:39.182922 23386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f1694b9e18cc46e7a3736331c8446d71" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:39.183339 23386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb73dffb61a47dca206831b250f5348 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:58:39.288868 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 23226
I20260620 01:58:39.313824 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 22963
I20260620 01:58:39.325133 23023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:58:39.463231 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 22963
I20260620 01:58:39.485158 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 22282
I20260620 01:58:39.488394 22345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:58:39.641580 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 22282
I20260620 01:58:39.645943 23375 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f1694b9e18cc46e7a3736331c8446d71" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8fb73dffb61a47dca206831b250f5348" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44173 } } peers { permanent_uuid: "fcd356f9391c4f2c806fb32328b70cd1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40165 } } peers { permanent_uuid: "f1694b9e18cc46e7a3736331c8446d71" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44741 } } peers { permanent_uuid: "3d5b20a61b4544f5a62dba46ae1b38d3" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 32825 } attrs { promote: false } } }
I20260620 01:58:39.646811 23375 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 [sys.catalog]: This master's current role is: LEADER
W20260620 01:58:39.647759 22372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.190:44173: connect: Connection refused (error 111)
W20260620 01:58:39.650851 22372 consensus_peers.cc:597] T 00000000000000000000000000000000 P f1694b9e18cc46e7a3736331c8446d71 -> Peer 8fb73dffb61a47dca206831b250f5348 (127.20.1.190:44173): Couldn't send request to peer 8fb73dffb61a47dca206831b250f5348. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:44173: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 01:58:39.668426 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 22352
I20260620 01:58:39.671367 22418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:58:39.807976 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 22352
I20260620 01:58:39.829142 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 22621
I20260620 01:58:39.831282 22793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:58:39.977638 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 22621
Shutting down hive metastore.
[ OK ] ExternalMiniClusterTest.TestAddMaster/3 (36896 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/4
2026-06-20T01:58:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:58:40Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/security.properties
2026-06-20 01:58:52: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260620 01:59:05.843276 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:41429
--webserver_interface=127.20.1.190
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955 with env {}
W20260620 01:59:06.373032 23683 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:06.373482 23683 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:06.373586 23683 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:06.383971 23683 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:59:06.384106 23683 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:06.384166 23683 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:59:06.384207 23683 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:59:06.398963 23683 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:41429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:06.401268 23683 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:06.403735 23683 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:06.417127 23688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:06.417151 23689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:06.417809 23691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:06.419876 23683 server_base.cc:1061] running on GCE node
I20260620 01:59:06.421175 23683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:06.423429 23683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:06.425027 23683 hybrid_clock.cc:648] HybridClock initialized: now 1781920746424919 us; error 54 us; skew 500 ppm
I20260620 01:59:06.426123 23683 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:06.429857 23683 webserver.cc:533] Webserver started at http://127.20.1.190:41241/ using document root <none> and password file <none>
I20260620 01:59:06.430972 23683 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:06.431164 23683 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:06.431643 23683 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:59:06.435559 23683 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "c70a600513004f9686fc2120acfc34fd"
format_stamp: "Formatted at 2026-06-20 01:59:06 on dist-test-slave-wg14"
I20260620 01:59:06.436666 23683 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "c70a600513004f9686fc2120acfc34fd"
format_stamp: "Formatted at 2026-06-20 01:59:06 on dist-test-slave-wg14"
I20260620 01:59:06.446000 23683 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.003s sys 0.005s
I20260620 01:59:06.453363 23697 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:06.456699 23683 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.004s sys 0.001s
I20260620 01:59:06.457252 23683 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "c70a600513004f9686fc2120acfc34fd"
format_stamp: "Formatted at 2026-06-20 01:59:06 on dist-test-slave-wg14"
I20260620 01:59:06.457885 23683 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:06.490074 23683 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:06.491312 23683 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:06.491664 23683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:06.521768 23683 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:41429
I20260620 01:59:06.521775 23748 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:41429 every 8 connection(s)
I20260620 01:59:06.524151 23683 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:59:06.525025 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 23683
I20260620 01:59:06.525274 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:59:06.527376 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:40235
--webserver_interface=127.20.1.189
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955 with env {}
I20260620 01:59:06.531580 23749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:59:06.540122 23749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:06.575585 23749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:59:06.577080 23699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:40235: connect: Connection refused (error 111)
W20260620 01:59:06.580741 23749 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:40235: Network error: Client connection negotiation failed: client connection to 127.20.1.189:40235: connect: Connection refused (error 111)
I20260620 01:59:06.630235 23749 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } attempt: 1
W20260620 01:59:06.633569 23749 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:40235: Network error: Client connection negotiation failed: client connection to 127.20.1.189:40235: connect: Connection refused (error 111)
I20260620 01:59:06.702044 23749 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } attempt: 2
W20260620 01:59:06.705165 23749 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:40235: Network error: Client connection negotiation failed: client connection to 127.20.1.189:40235: connect: Connection refused (error 111)
I20260620 01:59:06.796586 23749 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } attempt: 3
W20260620 01:59:06.799988 23749 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:40235: Network error: Client connection negotiation failed: client connection to 127.20.1.189:40235: connect: Connection refused (error 111)
I20260620 01:59:06.943325 23749 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } attempt: 4
W20260620 01:59:06.946365 23749 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:40235: Network error: Client connection negotiation failed: client connection to 127.20.1.189:40235: connect: Connection refused (error 111)
W20260620 01:59:07.018503 23751 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:07.018971 23751 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:07.019091 23751 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:07.033329 23751 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:59:07.033481 23751 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:07.033571 23751 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:59:07.033638 23751 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:59:07.046682 23751 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:40235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:07.048789 23751 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:07.051713 23751 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:07.064894 23759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:07.064916 23760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:07.065891 23751 server_base.cc:1061] running on GCE node
W20260620 01:59:07.066510 23762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:07.068575 23751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:07.071545 23751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:07.072913 23751 hybrid_clock.cc:648] HybridClock initialized: now 1781920747072822 us; error 84 us; skew 500 ppm
I20260620 01:59:07.073479 23751 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:07.076992 23751 webserver.cc:533] Webserver started at http://127.20.1.189:44857/ using document root <none> and password file <none>
I20260620 01:59:07.078089 23751 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:07.078263 23751 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:07.078687 23751 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:59:07.082849 23751 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "0b57d4ea494a4cabaf9dbc66570e4a41"
format_stamp: "Formatted at 2026-06-20 01:59:07 on dist-test-slave-wg14"
I20260620 01:59:07.085009 23751 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "0b57d4ea494a4cabaf9dbc66570e4a41"
format_stamp: "Formatted at 2026-06-20 01:59:07 on dist-test-slave-wg14"
I20260620 01:59:07.096254 23751 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.007s sys 0.004s
I20260620 01:59:07.104396 23768 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:07.107257 23751 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.005s sys 0.001s
I20260620 01:59:07.107577 23751 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "0b57d4ea494a4cabaf9dbc66570e4a41"
format_stamp: "Formatted at 2026-06-20 01:59:07 on dist-test-slave-wg14"
I20260620 01:59:07.107923 23751 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:07.132508 23751 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:07.134148 23751 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:07.134729 23751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:07.171224 23751 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:40235
I20260620 01:59:07.171262 23819 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:40235 every 8 connection(s)
I20260620 01:59:07.173454 23751 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:59:07.180176 23820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:59:07.181155 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 23751
I20260620 01:59:07.181463 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:59:07.182962 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:37955
--webserver_interface=127.20.1.188
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955 with env {}
I20260620 01:59:07.190357 23820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:07.214569 23820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:07.231889 23820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:59:07.234066 23769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37955: connect: Connection refused (error 111)
W20260620 01:59:07.236681 23820 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37955: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37955: connect: Connection refused (error 111)
I20260620 01:59:07.245764 23749 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } attempt: 5
I20260620 01:59:07.254640 23749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:59:07.258001 23749 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37955: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37955: connect: Connection refused (error 111)
I20260620 01:59:07.286381 23820 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } attempt: 1
W20260620 01:59:07.289923 23820 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37955: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37955: connect: Connection refused (error 111)
I20260620 01:59:07.309384 23749 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } attempt: 1
W20260620 01:59:07.313613 23749 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37955: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37955: connect: Connection refused (error 111)
I20260620 01:59:07.358305 23820 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } attempt: 2
W20260620 01:59:07.361299 23820 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37955: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37955: connect: Connection refused (error 111)
I20260620 01:59:07.382019 23749 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } attempt: 2
W20260620 01:59:07.385702 23749 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37955: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37955: connect: Connection refused (error 111)
I20260620 01:59:07.452678 23820 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } attempt: 3
W20260620 01:59:07.456207 23820 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37955: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37955: connect: Connection refused (error 111)
I20260620 01:59:07.492214 23749 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } attempt: 3
W20260620 01:59:07.494715 23749 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37955: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37955: connect: Connection refused (error 111)
I20260620 01:59:07.599682 23820 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } attempt: 4
W20260620 01:59:07.603917 23820 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37955: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37955: connect: Connection refused (error 111)
I20260620 01:59:07.672021 23749 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } attempt: 4
W20260620 01:59:07.675179 23749 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:37955: Network error: Client connection negotiation failed: client connection to 127.20.1.188:37955: connect: Connection refused (error 111)
W20260620 01:59:07.716325 23824 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:07.716719 23824 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:07.716813 23824 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:07.728423 23824 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:59:07.728565 23824 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:07.728626 23824 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:59:07.728667 23824 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:59:07.743564 23824 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:37955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:07.746613 23824 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:07.749894 23824 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:07.763018 23831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:07.765126 23832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:07.770421 23834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:07.771394 23824 server_base.cc:1061] running on GCE node
I20260620 01:59:07.773168 23824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:07.777508 23824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:07.779657 23824 hybrid_clock.cc:648] HybridClock initialized: now 1781920747779579 us; error 77 us; skew 500 ppm
I20260620 01:59:07.780221 23824 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:07.783726 23824 webserver.cc:533] Webserver started at http://127.20.1.188:38561/ using document root <none> and password file <none>
I20260620 01:59:07.784696 23824 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:07.784844 23824 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:07.785265 23824 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:59:07.788648 23824 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "349449c5d32240c184a5425e87ba8d1d"
format_stamp: "Formatted at 2026-06-20 01:59:07 on dist-test-slave-wg14"
I20260620 01:59:07.790911 23824 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "349449c5d32240c184a5425e87ba8d1d"
format_stamp: "Formatted at 2026-06-20 01:59:07 on dist-test-slave-wg14"
I20260620 01:59:07.801901 23824 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.009s sys 0.000s
I20260620 01:59:07.807802 23840 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:07.811733 23824 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260620 01:59:07.812028 23824 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "349449c5d32240c184a5425e87ba8d1d"
format_stamp: "Formatted at 2026-06-20 01:59:07 on dist-test-slave-wg14"
I20260620 01:59:07.812361 23824 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:07.844779 23824 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:07.845980 23824 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:07.846366 23824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:07.890820 23891 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:37955 every 8 connection(s)
I20260620 01:59:07.890865 23824 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:37955
I20260620 01:59:07.895596 23824 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:59:07.902071 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 23824
I20260620 01:59:07.902287 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:59:07.903245 23820 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } attempt: 5
I20260620 01:59:07.904604 23892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:59:07.918637 23892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:07.952598 23749 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } attempt: 5
I20260620 01:59:07.972117 23892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:07.983510 23820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41: Bootstrap starting.
I20260620 01:59:08.000070 23820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41: Neither blocks nor log segments found. Creating new log.
I20260620 01:59:08.001742 23820 log.cc:826] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41: Log is configured to *not* fsync() on all Append() calls
I20260620 01:59:08.004947 23820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41: No bootstrap required, opened a new log
I20260620 01:59:08.005435 23892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:08.013245 23749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd: Bootstrap starting.
I20260620 01:59:08.015811 23820 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } }
I20260620 01:59:08.016326 23820 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:59:08.016816 23820 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b57d4ea494a4cabaf9dbc66570e4a41, State: Initialized, Role: FOLLOWER
I20260620 01:59:08.017787 23820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } }
I20260620 01:59:08.021606 23749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd: Neither blocks nor log segments found. Creating new log.
I20260620 01:59:08.022054 23902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } }
I20260620 01:59:08.022815 23902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:08.025588 23820 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:59:08.027086 23892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d: Bootstrap starting.
I20260620 01:59:08.027547 23749 log.cc:826] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd: Log is configured to *not* fsync() on all Append() calls
I20260620 01:59:08.033560 23892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d: Neither blocks nor log segments found. Creating new log.
I20260620 01:59:08.036857 23892 log.cc:826] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d: Log is configured to *not* fsync() on all Append() calls
I20260620 01:59:08.038792 23749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd: No bootstrap required, opened a new log
I20260620 01:59:08.051440 23892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d: No bootstrap required, opened a new log
I20260620 01:59:08.052094 23749 raft_consensus.cc:359] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } }
I20260620 01:59:08.052779 23749 raft_consensus.cc:385] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:59:08.052990 23749 raft_consensus.cc:740] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c70a600513004f9686fc2120acfc34fd, State: Initialized, Role: FOLLOWER
I20260620 01:59:08.054517 23749 consensus_queue.cc:260] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } }
W20260620 01:59:08.058000 23914 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:59:08.058362 23914 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:59:08.043227 23751 master_runner.cc:187] Error getting master registration for 127.20.1.190:41429: OK, instance_id { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" instance_seqno: 1781920746513595 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:59:08.059084 23751 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:59:08.062273 23917 sys_catalog.cc:455] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } }
I20260620 01:59:08.062968 23917 sys_catalog.cc:458] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:08.062330 23892 raft_consensus.cc:359] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } }
I20260620 01:59:08.066263 23749 sys_catalog.cc:565] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:59:08.066507 23892 raft_consensus.cc:385] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:59:08.066932 23892 raft_consensus.cc:740] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349449c5d32240c184a5425e87ba8d1d, State: Initialized, Role: FOLLOWER
I20260620 01:59:08.071843 23892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } }
I20260620 01:59:08.073307 23919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } }
I20260620 01:59:08.073802 23919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:08.074838 23892 sys_catalog.cc:565] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:59:08.110100 23846 tablet.cc:2406] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 01:59:08.116040 23932 catalog_manager.cc:1593] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:59:08.116420 23932 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:59:08.130050 23902 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:59:08.131345 23902 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } }
I20260620 01:59:08.133821 23902 leader_election.cc:290] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c70a600513004f9686fc2120acfc34fd (127.20.1.190:41429), 349449c5d32240c184a5425e87ba8d1d (127.20.1.188:37955)
I20260620 01:59:08.135082 23724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c70a600513004f9686fc2120acfc34fd" is_pre_election: true
I20260620 01:59:08.135905 23724 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b57d4ea494a4cabaf9dbc66570e4a41 in term 0.
I20260620 01:59:08.136853 23770 leader_election.cc:304] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b57d4ea494a4cabaf9dbc66570e4a41, c70a600513004f9686fc2120acfc34fd; no voters:
I20260620 01:59:08.137699 23867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349449c5d32240c184a5425e87ba8d1d" is_pre_election: true
W20260620 01:59:08.139732 23774 tablet.cc:2406] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:59:08.139237 23867 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b57d4ea494a4cabaf9dbc66570e4a41 in term 0.
I20260620 01:59:08.150592 23902 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:59:08.151929 23902 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:59:08.152290 23902 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:59:08.157954 23902 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } }
I20260620 01:59:08.185393 23941 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 01:59:08.191314 23724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c70a600513004f9686fc2120acfc34fd"
I20260620 01:59:08.192248 23724 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:59:08.194860 23902 leader_election.cc:290] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [CANDIDATE]: Term 1 election: Requested vote from peers c70a600513004f9686fc2120acfc34fd (127.20.1.190:41429), 349449c5d32240c184a5425e87ba8d1d (127.20.1.188:37955)
I20260620 01:59:08.195914 23867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349449c5d32240c184a5425e87ba8d1d"
I20260620 01:59:08.196273 23867 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:59:08.199631 23724 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b57d4ea494a4cabaf9dbc66570e4a41 in term 1.
I20260620 01:59:08.201109 23770 leader_election.cc:304] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b57d4ea494a4cabaf9dbc66570e4a41, c70a600513004f9686fc2120acfc34fd; no voters:
I20260620 01:59:08.211047 23902 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:59:08.213665 23867 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b57d4ea494a4cabaf9dbc66570e4a41 in term 1.
I20260620 01:59:08.166430 23683 master_runner.cc:187] Error getting master registration for 127.20.1.188:37955: OK, instance_id { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" instance_seqno: 1781920747877617 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:59:08.216879 23683 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:59:08.219602 23902 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [term 1 LEADER]: Becoming Leader. State: Replica: 0b57d4ea494a4cabaf9dbc66570e4a41, State: Running, Role: LEADER
I20260620 01:59:08.221200 23902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } }
I20260620 01:59:08.243353 23942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b57d4ea494a4cabaf9dbc66570e4a41. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } }
I20260620 01:59:08.244997 23942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: This master's current role is: LEADER
W20260620 01:59:08.249742 23703 tablet.cc:2406] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:59:08.254493 23824 master_runner.cc:264] No leader master found from master 349449c5d32240c184a5425e87ba8d1d
I20260620 01:59:08.254743 23824 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:59:08.261212 23944 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:59:08.273819 23944 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:59:08.285714 23867 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [term 1 FOLLOWER]: Refusing update from remote peer 0b57d4ea494a4cabaf9dbc66570e4a41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:59:08.285722 23724 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [term 1 FOLLOWER]: Refusing update from remote peer 0b57d4ea494a4cabaf9dbc66570e4a41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:59:08.287251 23902 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:59:08.287767 23942 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:59:08.316033 23919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b57d4ea494a4cabaf9dbc66570e4a41. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } }
I20260620 01:59:08.316507 23919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:08.335705 23919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } }
I20260620 01:59:08.336072 23919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:08.339887 23942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } }
I20260620 01:59:08.340202 23942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:08.343299 23944 catalog_manager.cc:1382] Generated new cluster ID: 30232ade2a7c46ad90b591af4653b9e3
I20260620 01:59:08.343521 23944 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:59:08.348150 23917 sys_catalog.cc:455] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b57d4ea494a4cabaf9dbc66570e4a41. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } }
I20260620 01:59:08.348560 23917 sys_catalog.cc:458] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:08.350409 23902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } }
I20260620 01:59:08.350939 23902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:08.363929 23917 sys_catalog.cc:455] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } }
I20260620 01:59:08.364276 23917 sys_catalog.cc:458] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:08.406136 23944 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:59:08.407954 23944 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:59:08.435086 23944 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41: Generated new TSK 0
I20260620 01:59:08.436641 23944 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:59:08.457057 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955
--builtin_ntp_servers=127.20.1.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260620 01:59:08.999723 23954 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:09.000272 23954 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:09.000433 23954 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:09.014001 23954 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:09.014233 23954 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:59:09.029733 23954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:09.033748 23954 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:09.037473 23954 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:09.053478 23961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:09.053540 23960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:09.055781 23963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:09.056969 23954 server_base.cc:1061] running on GCE node
I20260620 01:59:09.058079 23954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:09.059808 23954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:09.063489 23954 hybrid_clock.cc:648] HybridClock initialized: now 1781920749063316 us; error 114 us; skew 500 ppm
I20260620 01:59:09.064172 23954 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:09.067485 23954 webserver.cc:533] Webserver started at http://127.20.1.129:34643/ using document root <none> and password file <none>
I20260620 01:59:09.068523 23954 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:09.068673 23954 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:09.069264 23954 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:59:09.074803 23954 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "caf149d35ae448f6abf3b13102049750"
format_stamp: "Formatted at 2026-06-20 01:59:09 on dist-test-slave-wg14"
I20260620 01:59:09.077059 23954 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "caf149d35ae448f6abf3b13102049750"
format_stamp: "Formatted at 2026-06-20 01:59:09 on dist-test-slave-wg14"
I20260620 01:59:09.087822 23954 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.006s sys 0.004s
I20260620 01:59:09.094069 23969 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:09.097442 23954 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260620 01:59:09.098161 23954 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "caf149d35ae448f6abf3b13102049750"
format_stamp: "Formatted at 2026-06-20 01:59:09 on dist-test-slave-wg14"
I20260620 01:59:09.098593 23954 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:09.119181 23932 catalog_manager.cc:1294] Loaded cluster ID: 30232ade2a7c46ad90b591af4653b9e3
I20260620 01:59:09.119334 23932 catalog_manager.cc:1587] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd: loading cluster ID for follower catalog manager: success
I20260620 01:59:09.122630 23954 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:09.123348 23932 catalog_manager.cc:1609] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd: acquiring CA information for follower catalog manager: success
I20260620 01:59:09.123899 23954 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:09.124214 23954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:09.125600 23954 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:59:09.126101 23932 catalog_manager.cc:1637] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:59:09.128433 23954 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:59:09.128599 23954 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:09.128733 23954 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:59:09.128818 23954 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:09.181074 23954 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38683
I20260620 01:59:09.181080 24081 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38683 every 8 connection(s)
I20260620 01:59:09.183919 23954 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:59:09.193838 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 23954
I20260620 01:59:09.194116 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:59:09.224627 23941 catalog_manager.cc:1294] Loaded cluster ID: 30232ade2a7c46ad90b591af4653b9e3
I20260620 01:59:09.224951 23941 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d: loading cluster ID for follower catalog manager: success
I20260620 01:59:09.240231 24083 heartbeater.cc:344] Connected to a master server at 127.20.1.190:41429
I20260620 01:59:09.240790 24083 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:09.243274 23941 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d: acquiring CA information for follower catalog manager: success
I20260620 01:59:09.243425 24083 heartbeater.cc:507] Master 127.20.1.190:41429 requested a full tablet report, sending...
I20260620 01:59:09.247159 24082 heartbeater.cc:344] Connected to a master server at 127.20.1.188:37955
I20260620 01:59:09.247435 24082 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:09.247584 23714 ts_manager.cc:194] Registered new tserver with Master: caf149d35ae448f6abf3b13102049750 (127.20.1.129:38683)
I20260620 01:59:09.248098 24082 heartbeater.cc:507] Master 127.20.1.188:37955 requested a full tablet report, sending...
I20260620 01:59:09.248126 23941 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:59:09.251173 23857 ts_manager.cc:194] Registered new tserver with Master: caf149d35ae448f6abf3b13102049750 (127.20.1.129:38683)
I20260620 01:59:09.256227 24084 heartbeater.cc:344] Connected to a master server at 127.20.1.189:40235
I20260620 01:59:09.256531 24084 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:09.257526 24084 heartbeater.cc:507] Master 127.20.1.189:40235 requested a full tablet report, sending...
I20260620 01:59:09.261147 23784 ts_manager.cc:194] Registered new tserver with Master: caf149d35ae448f6abf3b13102049750 (127.20.1.129:38683)
I20260620 01:59:09.263381 23784 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:59595
I20260620 01:59:09.274861 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:59:09.280820 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:34183
--webserver_interface=127.20.1.187
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955,127.20.1.187:34183 with env {}
W20260620 01:59:09.975679 24093 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:09.976279 24093 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:09.976409 24093 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:09.995426 24093 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:59:09.995586 24093 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:09.995669 24093 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:59:09.995736 24093 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:59:10.016968 24093 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955,127.20.1.187:34183
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:34183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.20.1.187
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:10.019191 24093 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:10.031380 24093 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:10.049759 24098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:10.049759 24099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:10.050745 24101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:10.052371 24093 server_base.cc:1061] running on GCE node
I20260620 01:59:10.054114 24093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:10.056632 24093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:10.058019 24093 hybrid_clock.cc:648] HybridClock initialized: now 1781920750057956 us; error 85 us; skew 500 ppm
I20260620 01:59:10.058485 24093 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:10.063715 24093 webserver.cc:533] Webserver started at http://127.20.1.187:33031/ using document root <none> and password file <none>
I20260620 01:59:10.064868 24093 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:10.065063 24093 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:10.065506 24093 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:59:10.068867 24093 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/data/instance:
uuid: "0b78bdb32e4f411b80a16bac09c1f2f7"
format_stamp: "Formatted at 2026-06-20 01:59:10 on dist-test-slave-wg14"
I20260620 01:59:10.070750 24093 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/wal/instance:
uuid: "0b78bdb32e4f411b80a16bac09c1f2f7"
format_stamp: "Formatted at 2026-06-20 01:59:10 on dist-test-slave-wg14"
I20260620 01:59:10.079478 24093 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s
I20260620 01:59:10.090785 24107 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:10.093854 24093 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.007s sys 0.004s
I20260620 01:59:10.094318 24093 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "0b78bdb32e4f411b80a16bac09c1f2f7"
format_stamp: "Formatted at 2026-06-20 01:59:10 on dist-test-slave-wg14"
I20260620 01:59:10.094648 24093 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:10.136432 24093 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:10.137874 24093 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:10.138240 24093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:10.176246 24093 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:34183
I20260620 01:59:10.176388 24158 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:34183 every 8 connection(s)
I20260620 01:59:10.179080 24093 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 01:59:10.181203 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 24093
I20260620 01:59:10.194761 24159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:59:10.212186 24159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:10.237636 24159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:10.247957 24159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:10.260052 24159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:10.270164 24084 heartbeater.cc:499] Master 127.20.1.189:40235 was elected leader, sending a full tablet report...
I20260620 01:59:10.281864 24159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: Bootstrap starting.
I20260620 01:59:10.286446 24159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: Neither blocks nor log segments found. Creating new log.
I20260620 01:59:10.288427 24159 log.cc:826] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: Log is configured to *not* fsync() on all Append() calls
I20260620 01:59:10.293251 24159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: No bootstrap required, opened a new log
I20260620 01:59:10.303048 24159 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } }
I20260620 01:59:10.303709 24159 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:59:10.303869 24159 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b78bdb32e4f411b80a16bac09c1f2f7, State: Initialized, Role: FOLLOWER
I20260620 01:59:10.304730 24159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } }
I20260620 01:59:10.306596 24168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } } }
I20260620 01:59:10.307389 24168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:10.307696 24159 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:59:10.330169 24179 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:59:10.330485 24179 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:59:10.344906 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 23954
I20260620 01:59:10.345898 24093 master_runner.cc:429] Detected that this master 0b78bdb32e4f411b80a16bac09c1f2f7 is joining an existing cluster
I20260620 01:59:10.346107 24093 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260620 01:59:10.348922 24093 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 01:59:10.349707 24093 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 01:59:10.350181 24093 tablet_replica.cc:333] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: stopping tablet replica
I20260620 01:59:10.353098 24093 master_runner.cc:434] Initiating AddMaster RPC to add 127.20.1.187:34183
I20260620 01:59:10.371724 24077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:10.458792 23784 catalog_manager.cc:7201] Initiating ChangeConfig request to add master 127.20.1.187:34183: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7"
member_type: NON_VOTER
last_known_addr {
host: "127.20.1.187"
port: 34183
}
attrs {
promote: true
}
}
dest_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41"
cas_config_opid_index: -1
I20260620 01:59:10.460664 23784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: true } }
I20260620 01:59:10.466105 23867 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [term 1 FOLLOWER]: Refusing update from remote peer 0b57d4ea494a4cabaf9dbc66570e4a41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 01:59:10.466246 23724 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [term 1 FOLLOWER]: Refusing update from remote peer 0b57d4ea494a4cabaf9dbc66570e4a41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 01:59:10.467234 23955 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:59:10.468269 23955 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260620 01:59:10.467061 23772 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 -> Peer 0b78bdb32e4f411b80a16bac09c1f2f7 (127.20.1.187:34183): Couldn't send request to peer 0b78bdb32e4f411b80a16bac09c1f2f7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260620 01:59:10.481837 23955 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0b78bdb32e4f411b80a16bac09c1f2f7 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: true } } }
I20260620 01:59:10.490152 23867 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0b78bdb32e4f411b80a16bac09c1f2f7 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: true } } }
I20260620 01:59:10.497614 23955 catalog_manager.cc:7205] Successfully completed master ChangeConfig request to add master 127.20.1.187:34183
I20260620 01:59:10.498790 24093 master.cc:562] Master@127.20.1.187:34183 shutting down...
I20260620 01:59:10.506069 24199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: true } } }
I20260620 01:59:10.506897 24199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:10.506783 23724 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0b78bdb32e4f411b80a16bac09c1f2f7 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: true } } }
I20260620 01:59:10.507372 24196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: true } } }
I20260620 01:59:10.507869 24196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:10.510226 24196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: true } } }
I20260620 01:59:10.510623 24196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:10.515477 24198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: true } } }
I20260620 01:59:10.515995 24198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:10.517256 24201 sys_catalog.cc:455] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: true } } }
I20260620 01:59:10.517577 24201 sys_catalog.cc:458] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:10.551244 24093 master.cc:584] Master@127.20.1.187:34183 shutdown complete.
I20260620 01:59:10.551791 24093 master_runner.cc:306] Clearing existing system tablet
I20260620 01:59:10.552284 24093 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:10.552418 24093 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:10.555198 24093 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 01:59:10.558139 24203 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:10.558647 24093 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:10.558939 24093 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 01:59:10.559073 24093 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "0b78bdb32e4f411b80a16bac09c1f2f7"
format_stamp: "Formatted at 2026-06-20 01:59:10 on dist-test-slave-wg14"
I20260620 01:59:10.565145 24093 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 01:59:10.576104 24093 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 01:59:10.576361 24093 log.cc:1199] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: Deleting WAL directory at /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260620 01:59:10.577188 24093 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: Deleting consensus metadata
I20260620 01:59:10.577492 24093 master_runner.cc:316] Copying system tablet from 127.20.1.189:40235
I20260620 01:59:10.584438 24093 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.1.189:40235
I20260620 01:59:10.594592 23805 tablet_copy_service.cc:140] P 0b57d4ea494a4cabaf9dbc66570e4a41: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0b78bdb32e4f411b80a16bac09c1f2f7 ({username='slave'} at 127.0.0.1:54522)
I20260620 01:59:10.595445 23805 tablet_copy_service.cc:161] P 0b57d4ea494a4cabaf9dbc66570e4a41: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0b78bdb32e4f411b80a16bac09c1f2f7 at {username='slave'} at 127.0.0.1:54522: session id = 0b78bdb32e4f411b80a16bac09c1f2f7-00000000000000000000000000000000
I20260620 01:59:10.601456 23805 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 01:59:10.604768 24093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 01:59:10.610961 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 23954
I20260620 01:59:10.614719 24093 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: tablet copy: Starting download of 0 data blocks...
I20260620 01:59:10.615191 24093 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: tablet copy: Starting download of 1 WAL segments...
I20260620 01:59:10.620429 24093 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 01:59:10.632136 23805 tablet_copy_service.cc:342] P 0b57d4ea494a4cabaf9dbc66570e4a41: Request end of tablet copy session 0b78bdb32e4f411b80a16bac09c1f2f7-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:54522
I20260620 01:59:10.632664 23805 tablet_copy_service.cc:434] P 0b57d4ea494a4cabaf9dbc66570e4a41: ending tablet copy session 0b78bdb32e4f411b80a16bac09c1f2f7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0b78bdb32e4f411b80a16bac09c1f2f7
I20260620 01:59:10.637894 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:38683
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955
--webserver_port=34643
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955,127.20.1.187:34183 with env {}
W20260620 01:59:10.652900 24093 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260620 01:59:10.657756 24093 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:10.661073 24210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:10.668098 24211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:10.679452 24214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:10.693504 24093 server_base.cc:1061] running on GCE node
I20260620 01:59:10.694432 24093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:10.695402 24093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:10.696802 24093 hybrid_clock.cc:648] HybridClock initialized: now 1781920750696732 us; error 48 us; skew 500 ppm
I20260620 01:59:10.697192 24093 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:10.699831 24093 webserver.cc:533] Webserver started at http://127.20.1.187:38119/ using document root <none> and password file <none>
I20260620 01:59:10.702898 24093 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:10.704207 24093 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:10.710287 24093 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 01:59:10.714030 24220 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:10.714964 24093 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 01:59:10.715256 24093 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "0b78bdb32e4f411b80a16bac09c1f2f7"
format_stamp: "Formatted at 2026-06-20 01:59:10 on dist-test-slave-wg14"
I20260620 01:59:10.715461 24093 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:10.777336 24093 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:10.778824 24093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:10.882580 24093 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:34183
I20260620 01:59:10.882755 24271 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:34183 every 8 connection(s)
I20260620 01:59:10.884460 24093 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 01:59:10.893239 24272 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:59:10.897954 24272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: Bootstrap starting.
I20260620 01:59:10.932371 24272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:59:10.933213 24272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: Bootstrap complete.
I20260620 01:59:10.935714 24272 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: true } }
I20260620 01:59:10.937526 24272 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0b78bdb32e4f411b80a16bac09c1f2f7, State: Initialized, Role: LEARNER
I20260620 01:59:10.938263 24272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: true } }
I20260620 01:59:10.939905 24275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: true } } }
I20260620 01:59:10.940385 24272 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:59:10.940562 24275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [sys.catalog]: This master's current role is: LEARNER
I20260620 01:59:10.968708 24286 catalog_manager.cc:1294] Loaded cluster ID: 30232ade2a7c46ad90b591af4653b9e3
I20260620 01:59:10.968915 24286 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: loading cluster ID for follower catalog manager: success
I20260620 01:59:10.977072 24286 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: acquiring CA information for follower catalog manager: success
I20260620 01:59:10.982795 24286 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:59:11.082667 24247 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260620 01:59:11.084568 24275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b57d4ea494a4cabaf9dbc66570e4a41. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: true } } }
I20260620 01:59:11.084956 24275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [sys.catalog]: This master's current role is: LEARNER
W20260620 01:59:11.488793 24212 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:11.489233 24212 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:11.489346 24212 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:11.503983 24212 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:11.504686 24212 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:59:11.504598 24199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: true } } }
I20260620 01:59:11.505041 24199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:11.508165 24293 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41: attempting to promote NON_VOTER 0b78bdb32e4f411b80a16bac09c1f2f7 to VOTER
I20260620 01:59:11.509548 24293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } }
I20260620 01:59:11.513175 24247 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [term 1 LEARNER]: Refusing update from remote peer 0b57d4ea494a4cabaf9dbc66570e4a41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 01:59:11.514081 23867 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [term 1 FOLLOWER]: Refusing update from remote peer 0b57d4ea494a4cabaf9dbc66570e4a41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 01:59:11.514117 23724 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [term 1 FOLLOWER]: Refusing update from remote peer 0b57d4ea494a4cabaf9dbc66570e4a41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 01:59:11.514271 24296 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:59:11.515499 24199 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:59:11.515815 24292 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:59:11.525231 24212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:38683
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=34643
--tserver_master_addrs=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955,127.20.1.187:34183
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:11.528647 24212 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:11.533008 24212 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260620 01:59:11.542424 24296 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0b78bdb32e4f411b80a16bac09c1f2f7 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } } }
I20260620 01:59:11.544098 24247 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0b78bdb32e4f411b80a16bac09c1f2f7 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } } }
I20260620 01:59:11.549862 23867 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0b78bdb32e4f411b80a16bac09c1f2f7 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } } }
I20260620 01:59:11.551224 24275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } } }
I20260620 01:59:11.552173 24275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b78bdb32e4f411b80a16bac09c1f2f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:11.552497 23724 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0b78bdb32e4f411b80a16bac09c1f2f7 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } } }
I20260620 01:59:11.557322 24292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } } }
I20260620 01:59:11.557766 24292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:11.558068 24301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } } }
I20260620 01:59:11.558624 24301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 349449c5d32240c184a5425e87ba8d1d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:11.558535 24292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } } }
I20260620 01:59:11.558794 24292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:11.559242 24292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } } }
I20260620 01:59:11.559459 24292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:11.559551 24302 sys_catalog.cc:455] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } } }
I20260620 01:59:11.560317 24302 sys_catalog.cc:458] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:11.561551 24292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } } }
I20260620 01:59:11.563081 24292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: This master's current role is: LEADER
W20260620 01:59:11.571895 24303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:11.573001 24304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:11.574780 24306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:11.575587 24212 server_base.cc:1061] running on GCE node
I20260620 01:59:11.576797 24212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:11.578594 24212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:11.580534 24212 hybrid_clock.cc:648] HybridClock initialized: now 1781920751580282 us; error 106 us; skew 500 ppm
I20260620 01:59:11.581413 24212 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:11.590011 24212 webserver.cc:533] Webserver started at http://127.20.1.129:34643/ using document root <none> and password file <none>
I20260620 01:59:11.591274 24212 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:11.591459 24212 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:11.599737 24212 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 01:59:11.609560 24312 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:11.612156 24212 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
I20260620 01:59:11.612459 24212 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "caf149d35ae448f6abf3b13102049750"
format_stamp: "Formatted at 2026-06-20 01:59:09 on dist-test-slave-wg14"
I20260620 01:59:11.613520 24212 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:11.651737 24212 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:11.653337 24212 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:11.653723 24212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:11.655673 24212 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:59:11.658072 24212 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:59:11.658200 24212 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:11.658306 24212 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:59:11.658352 24212 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:11.734946 24212 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38683
I20260620 01:59:11.734999 24424 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38683 every 8 connection(s)
I20260620 01:59:11.737001 24212 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:59:11.738014 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 24212
I20260620 01:59:11.809350 24425 heartbeater.cc:344] Connected to a master server at 127.20.1.188:37955
I20260620 01:59:11.810267 24427 heartbeater.cc:344] Connected to a master server at 127.20.1.190:41429
I20260620 01:59:11.810426 24425 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:11.810539 24427 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:11.813015 24425 heartbeater.cc:507] Master 127.20.1.188:37955 requested a full tablet report, sending...
I20260620 01:59:11.813184 24427 heartbeater.cc:507] Master 127.20.1.190:41429 requested a full tablet report, sending...
I20260620 01:59:11.815230 24426 heartbeater.cc:344] Connected to a master server at 127.20.1.187:34183
I20260620 01:59:11.815454 24426 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:11.816038 24426 heartbeater.cc:507] Master 127.20.1.187:34183 requested a full tablet report, sending...
I20260620 01:59:11.816056 23857 ts_manager.cc:194] Re-registered known tserver with Master: caf149d35ae448f6abf3b13102049750 (127.20.1.129:38683)
I20260620 01:59:11.816874 23714 ts_manager.cc:194] Re-registered known tserver with Master: caf149d35ae448f6abf3b13102049750 (127.20.1.129:38683)
I20260620 01:59:11.818544 24237 ts_manager.cc:194] Registered new tserver with Master: caf149d35ae448f6abf3b13102049750 (127.20.1.129:38683)
I20260620 01:59:11.823992 24434 heartbeater.cc:344] Connected to a master server at 127.20.1.189:40235
I20260620 01:59:11.824468 24434 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:11.825199 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:59:11.825932 24434 heartbeater.cc:507] Master 127.20.1.189:40235 requested a full tablet report, sending...
I20260620 01:59:11.828548 23784 ts_manager.cc:194] Re-registered known tserver with Master: caf149d35ae448f6abf3b13102049750 (127.20.1.129:38683)
I20260620 01:59:11.829857 23784 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:44173
I20260620 01:59:11.848829 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:59:11.849164 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 23683
I20260620 01:59:11.852099 23744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:12.020645 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 23683
I20260620 01:59:12.053148 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:41429
--webserver_interface=127.20.1.190
--webserver_port=41241
--builtin_ntp_servers=127.20.1.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955
--master_addresses=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955,127.20.1.187:34183 with env {}
W20260620 01:59:12.153049 23770 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 -> Peer c70a600513004f9686fc2120acfc34fd (127.20.1.190:41429): Couldn't send request to peer c70a600513004f9686fc2120acfc34fd. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:41429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 01:59:12.632177 24292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } } }
I20260620 01:59:12.632593 24292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: This master's current role is: LEADER
W20260620 01:59:12.635437 23770 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.190:41429: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20260620 01:59:12.727069 24441 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:12.727626 24441 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:12.727782 24441 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:12.749163 24441 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:59:12.749496 24441 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:12.749620 24441 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:59:12.750092 24441 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:59:12.767345 24441 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955,127.20.1.187:34183
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:41429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=41241
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:12.769455 24441 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:12.772814 24441 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:12.795874 24451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:12.795851 24452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:12.795882 24454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:12.798956 24441 server_base.cc:1061] running on GCE node
I20260620 01:59:12.800362 24441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:12.804791 24441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:12.808575 24441 hybrid_clock.cc:648] HybridClock initialized: now 1781920752808272 us; error 265 us; skew 500 ppm
I20260620 01:59:12.809437 24441 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:12.813930 24441 webserver.cc:533] Webserver started at http://127.20.1.190:41241/ using document root <none> and password file <none>
I20260620 01:59:12.815747 24441 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:12.815984 24441 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260620 01:59:12.826074 24427 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:41429 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:41429: connect: Connection refused (error 111)
I20260620 01:59:12.830529 24441 fs_manager.cc:714] Time spent opening directory manager: real 0.012s user 0.007s sys 0.004s
I20260620 01:59:12.834687 24434 heartbeater.cc:499] Master 127.20.1.189:40235 was elected leader, sending a full tablet report...
I20260620 01:59:12.835772 24461 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:12.839542 24441 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.004s sys 0.001s
I20260620 01:59:12.839847 24441 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "c70a600513004f9686fc2120acfc34fd"
format_stamp: "Formatted at 2026-06-20 01:59:06 on dist-test-slave-wg14"
I20260620 01:59:12.841152 24441 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:12.874037 24441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:12.875249 24441 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:12.875701 24441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:12.927873 24441 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:41429
I20260620 01:59:12.927872 24512 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:41429 every 8 connection(s)
I20260620 01:59:12.931869 24441 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:59:12.935699 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 24441
I20260620 01:59:12.936287 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 24212
I20260620 01:59:12.946938 24513 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:59:12.958452 24513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd: Bootstrap starting.
I20260620 01:59:12.961426 24420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:12.993209 24513 log.cc:826] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd: Log is configured to *not* fsync() on all Append() calls
I20260620 01:59:13.010291 24513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:59:13.012051 24513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd: Bootstrap complete.
I20260620 01:59:13.026603 24513 raft_consensus.cc:359] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } }
I20260620 01:59:13.027725 24513 raft_consensus.cc:740] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c70a600513004f9686fc2120acfc34fd, State: Initialized, Role: FOLLOWER
I20260620 01:59:13.029501 24513 consensus_queue.cc:260] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } }
I20260620 01:59:13.032266 24519 sys_catalog.cc:455] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } } }
I20260620 01:59:13.033100 24519 sys_catalog.cc:458] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:13.035104 24513 sys_catalog.cc:565] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:59:13.074568 24530 catalog_manager.cc:1294] Loaded cluster ID: 30232ade2a7c46ad90b591af4653b9e3
I20260620 01:59:13.074723 24530 catalog_manager.cc:1587] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd: loading cluster ID for follower catalog manager: success
I20260620 01:59:13.095969 24530 catalog_manager.cc:1609] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd: acquiring CA information for follower catalog manager: success
I20260620 01:59:13.106118 24530 catalog_manager.cc:1637] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:59:13.151723 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 24212
I20260620 01:59:13.199218 24519 sys_catalog.cc:455] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b57d4ea494a4cabaf9dbc66570e4a41. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } } }
I20260620 01:59:13.199568 24519 sys_catalog.cc:458] T 00000000000000000000000000000000 P c70a600513004f9686fc2120acfc34fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:13.217890 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:38683
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955
--webserver_port=34643
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955,127.20.1.187:34183 with env {}
I20260620 01:59:13.535436 24445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c70a600513004f9686fc2120acfc34fd" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41429 } } peers { permanent_uuid: "0b57d4ea494a4cabaf9dbc66570e4a41" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 40235 } } peers { permanent_uuid: "349449c5d32240c184a5425e87ba8d1d" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 37955 } } peers { permanent_uuid: "0b78bdb32e4f411b80a16bac09c1f2f7" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 34183 } attrs { promote: false } } }
I20260620 01:59:13.537827 24445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b57d4ea494a4cabaf9dbc66570e4a41 [sys.catalog]: This master's current role is: LEADER
W20260620 01:59:13.869314 24537 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:13.870160 24537 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:13.870464 24537 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:13.883956 24537 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:13.884227 24537 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:59:13.897130 24537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:38683
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=34643
--tserver_master_addrs=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955,127.20.1.187:34183
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:13.899528 24537 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:13.902019 24537 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:13.916491 24549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:13.916291 24551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:13.916323 24548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:13.919247 24537 server_base.cc:1061] running on GCE node
I20260620 01:59:13.920858 24537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:13.922748 24537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:13.924129 24537 hybrid_clock.cc:648] HybridClock initialized: now 1781920753924037 us; error 96 us; skew 500 ppm
I20260620 01:59:13.924680 24537 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:13.927615 24537 webserver.cc:533] Webserver started at http://127.20.1.129:34643/ using document root <none> and password file <none>
I20260620 01:59:13.928689 24537 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:13.928843 24537 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:13.938596 24537 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:59:13.944175 24557 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:13.947490 24537 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260620 01:59:13.947778 24537 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "caf149d35ae448f6abf3b13102049750"
format_stamp: "Formatted at 2026-06-20 01:59:09 on dist-test-slave-wg14"
I20260620 01:59:13.948818 24537 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:14.000288 24537 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:14.001796 24537 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:14.002188 24537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:14.003520 24537 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:59:14.005853 24537 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:59:14.005968 24537 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:14.006088 24537 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:59:14.006137 24537 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:14.057544 24537 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38683
I20260620 01:59:14.057668 24669 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38683 every 8 connection(s)
I20260620 01:59:14.059576 24537 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:59:14.068557 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 24537
I20260620 01:59:14.131649 24671 heartbeater.cc:344] Connected to a master server at 127.20.1.187:34183
I20260620 01:59:14.132202 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 24537
I20260620 01:59:14.132241 24671 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:14.133718 24671 heartbeater.cc:507] Master 127.20.1.187:34183 requested a full tablet report, sending...
I20260620 01:59:14.137820 24237 ts_manager.cc:194] Re-registered known tserver with Master: caf149d35ae448f6abf3b13102049750 (127.20.1.129:38683)
I20260620 01:59:14.141377 24670 heartbeater.cc:344] Connected to a master server at 127.20.1.188:37955
I20260620 01:59:14.141589 24670 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:14.142410 24670 heartbeater.cc:507] Master 127.20.1.188:37955 requested a full tablet report, sending...
I20260620 01:59:14.144811 23857 ts_manager.cc:194] Re-registered known tserver with Master: caf149d35ae448f6abf3b13102049750 (127.20.1.129:38683)
I20260620 01:59:14.149108 24672 heartbeater.cc:344] Connected to a master server at 127.20.1.190:41429
I20260620 01:59:14.149351 24672 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:14.150007 24672 heartbeater.cc:507] Master 127.20.1.190:41429 requested a full tablet report, sending...
I20260620 01:59:14.152868 24477 ts_manager.cc:194] Registered new tserver with Master: caf149d35ae448f6abf3b13102049750 (127.20.1.129:38683)
I20260620 01:59:14.157421 24673 heartbeater.cc:344] Connected to a master server at 127.20.1.189:40235
I20260620 01:59:14.157661 24673 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:14.158203 24673 heartbeater.cc:507] Master 127.20.1.189:40235 requested a full tablet report, sending...
I20260620 01:59:14.159646 23784 ts_manager.cc:194] Re-registered known tserver with Master: caf149d35ae448f6abf3b13102049750 (127.20.1.129:38683)
I20260620 01:59:14.160885 23784 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:56903
I20260620 01:59:14.164530 24665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:14.272462 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 24537
I20260620 01:59:14.290369 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:38683
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955
--webserver_port=34643
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955,127.20.1.187:34183
--fault_before_start=1.0 with env {}
W20260620 01:59:14.786695 24688 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:14.787525 24688 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:14.787717 24688 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:59:14.787870 24688 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:14.800827 24688 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:14.801116 24688 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:59:14.815618 24688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:38683
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=34643
--tserver_master_addrs=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955,127.20.1.187:34183
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:14.817610 24688 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:14.820528 24688 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:14.837229 24693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:14.838656 24694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:14.839635 24696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:14.841483 24688 server_base.cc:1061] running on GCE node
I20260620 01:59:14.842423 24688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:14.844156 24688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:14.847745 24688 hybrid_clock.cc:648] HybridClock initialized: now 1781920754847529 us; error 125 us; skew 500 ppm
I20260620 01:59:14.848258 24688 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:14.851902 24688 webserver.cc:533] Webserver started at http://127.20.1.129:34643/ using document root <none> and password file <none>
I20260620 01:59:14.852911 24688 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:14.853093 24688 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:14.861181 24688 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:59:14.866413 24702 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:14.869154 24688 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.003s
I20260620 01:59:14.869724 24688 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "caf149d35ae448f6abf3b13102049750"
format_stamp: "Formatted at 2026-06-20 01:59:09 on dist-test-slave-wg14"
I20260620 01:59:14.871570 24688 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:14.921074 24688 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:14.922761 24688 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:14.923098 24688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:14.924590 24688 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:59:14.927006 24688 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:59:14.927194 24688 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:14.927353 24688 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:59:14.927438 24688 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:59:14.927999 24688 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:59:14.928120 24688 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:59:14.928254 24688 fault_injection.cc:59] injecting fault for kudu (pid 24688): FLAGS_fault_before_start (process will exit)
I20260620 01:59:14.952556 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:38683
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955
--webserver_port=34643
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955,127.20.1.187:34183 with env {}
W20260620 01:59:15.576696 24710 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:15.577081 24710 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:15.577198 24710 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:15.593344 24710 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:15.593560 24710 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:59:15.612356 24710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:38683
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=34643
--tserver_master_addrs=127.20.1.190:41429,127.20.1.189:40235,127.20.1.188:37955,127.20.1.187:34183
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:15.614967 24710 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:15.619884 24710 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:15.638013 24715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:15.639614 24716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:15.643183 24718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:15.644411 24710 server_base.cc:1061] running on GCE node
I20260620 01:59:15.646862 24710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:15.648774 24710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:15.652340 24710 hybrid_clock.cc:648] HybridClock initialized: now 1781920755652326 us; error 250 us; skew 500 ppm
I20260620 01:59:15.652974 24710 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:15.657079 24710 webserver.cc:533] Webserver started at http://127.20.1.129:34643/ using document root <none> and password file <none>
I20260620 01:59:15.658178 24710 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:15.658377 24710 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:15.677531 24710 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 01:59:15.692260 24724 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:15.694749 24710 fs_manager.cc:730] Time spent opening block manager: real 0.015s user 0.009s sys 0.000s
I20260620 01:59:15.695070 24710 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "caf149d35ae448f6abf3b13102049750"
format_stamp: "Formatted at 2026-06-20 01:59:09 on dist-test-slave-wg14"
I20260620 01:59:15.696369 24710 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:15.772989 24710 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:15.774191 24710 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:15.774578 24710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:15.777977 24710 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:59:15.792074 24710 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:59:15.792295 24710 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:15.792515 24710 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:59:15.792601 24710 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:15.917640 24710 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38683
I20260620 01:59:15.917701 24836 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38683 every 8 connection(s)
I20260620 01:59:15.919919 24710 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:59:15.930981 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 24710
I20260620 01:59:15.931386 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 24710
I20260620 01:59:15.977134 24838 heartbeater.cc:344] Connected to a master server at 127.20.1.187:34183
I20260620 01:59:15.977666 24839 heartbeater.cc:344] Connected to a master server at 127.20.1.190:41429
I20260620 01:59:15.978492 24838 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:15.978513 24839 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:15.988315 24839 heartbeater.cc:507] Master 127.20.1.190:41429 requested a full tablet report, sending...
I20260620 01:59:15.995983 24477 ts_manager.cc:194] Re-registered known tserver with Master: caf149d35ae448f6abf3b13102049750 (127.20.1.129:38683)
I20260620 01:59:15.999236 24838 heartbeater.cc:507] Master 127.20.1.187:34183 requested a full tablet report, sending...
I20260620 01:59:16.001631 24237 ts_manager.cc:194] Re-registered known tserver with Master: caf149d35ae448f6abf3b13102049750 (127.20.1.129:38683)
I20260620 01:59:16.013324 24837 heartbeater.cc:344] Connected to a master server at 127.20.1.188:37955
I20260620 01:59:16.013613 24837 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:16.014581 24837 heartbeater.cc:507] Master 127.20.1.188:37955 requested a full tablet report, sending...
I20260620 01:59:16.016129 24832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:16.016451 23857 ts_manager.cc:194] Re-registered known tserver with Master: caf149d35ae448f6abf3b13102049750 (127.20.1.129:38683)
I20260620 01:59:16.203207 24840 heartbeater.cc:344] Connected to a master server at 127.20.1.189:40235
I20260620 01:59:16.203446 24840 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:16.203987 24840 heartbeater.cc:507] Master 127.20.1.189:40235 requested a full tablet report, sending...
I20260620 01:59:16.205578 23784 ts_manager.cc:194] Re-registered known tserver with Master: caf149d35ae448f6abf3b13102049750 (127.20.1.129:38683)
I20260620 01:59:16.206663 23784 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:41709
I20260620 01:59:16.207412 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 24710
I20260620 01:59:16.227538 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 24441
W20260620 01:59:16.227830 23772 connection.cc:573] server connection from 127.20.1.129:41709 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:59:16.239404 24508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:16.397912 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 24441
I20260620 01:59:16.424232 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 23751
I20260620 01:59:16.426523 23815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:16.603832 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 23751
W20260620 01:59:16.637555 23844 connection.cc:573] server connection from 127.0.0.1:39736 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:59:16.637974 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 23824
I20260620 01:59:16.639410 23887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:16.829612 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 23824
I20260620 01:59:16.870517 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 24093
I20260620 01:59:16.872937 24267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:17.024904 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 24093
Shutting down hive metastore.
2026-06-20T01:59:17Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/4 (37088 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/5
2026-06-20T01:59:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:59:17Z Disabled control of system clock
2026-06-20T01:59:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:59:17Z Disabled control of system clock
2026-06-20T01:59:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:59:17Z Disabled control of system clock
2026-06-20T01:59:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:59:17Z Disabled control of system clock
2026-06-20T01:59:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T01:59:17Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/security.properties
2026-06-20 01:59:30: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260620 01:59:43.530196 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42763
--webserver_interface=127.20.1.190
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969 with env {}
W20260620 01:59:43.976380 25156 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:43.977172 25156 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:43.977347 25156 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:43.987943 25156 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:59:43.988090 25156 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:43.988157 25156 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:59:43.988201 25156 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:59:44.001621 25156 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:44.004318 25156 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:44.007117 25156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:44.019574 25161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:44.019665 25164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:44.019639 25162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:44.021812 25156 server_base.cc:1061] running on GCE node
I20260620 01:59:44.023119 25156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:44.025583 25156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:44.029253 25156 hybrid_clock.cc:648] HybridClock initialized: now 1781920784028973 us; error 189 us; skew 500 ppm
I20260620 01:59:44.030349 25156 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:44.034906 25156 webserver.cc:533] Webserver started at http://127.20.1.190:34373/ using document root <none> and password file <none>
I20260620 01:59:44.036000 25156 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:44.036144 25156 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:44.036569 25156 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:59:44.039711 25156 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "1f88aeb256ed4b37aa42049861c6bbc2"
format_stamp: "Formatted at 2026-06-20 01:59:44 on dist-test-slave-wg14"
I20260620 01:59:44.040807 25156 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "1f88aeb256ed4b37aa42049861c6bbc2"
format_stamp: "Formatted at 2026-06-20 01:59:44 on dist-test-slave-wg14"
I20260620 01:59:44.049264 25156 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260620 01:59:44.054775 25170 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:44.057252 25156 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:59:44.057726 25156 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "1f88aeb256ed4b37aa42049861c6bbc2"
format_stamp: "Formatted at 2026-06-20 01:59:44 on dist-test-slave-wg14"
I20260620 01:59:44.058104 25156 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:44.081804 25156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:44.083388 25156 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:44.083766 25156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:44.110932 25156 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:42763
I20260620 01:59:44.110906 25221 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:42763 every 8 connection(s)
I20260620 01:59:44.113093 25156 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:59:44.119436 25222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:59:44.120100 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 25156
I20260620 01:59:44.120321 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:59:44.121773 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:38329
--webserver_interface=127.20.1.189
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969 with env {}
I20260620 01:59:44.127584 25222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:44.150880 25222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:59:44.153189 25173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:38329: connect: Connection refused (error 111)
W20260620 01:59:44.156478 25222 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:38329: Network error: Client connection negotiation failed: client connection to 127.20.1.189:38329: connect: Connection refused (error 111)
I20260620 01:59:44.206025 25222 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } attempt: 1
W20260620 01:59:44.208659 25222 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:38329: Network error: Client connection negotiation failed: client connection to 127.20.1.189:38329: connect: Connection refused (error 111)
I20260620 01:59:44.277055 25222 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } attempt: 2
W20260620 01:59:44.279621 25222 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:38329: Network error: Client connection negotiation failed: client connection to 127.20.1.189:38329: connect: Connection refused (error 111)
I20260620 01:59:44.371021 25222 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } attempt: 3
W20260620 01:59:44.373534 25222 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:38329: Network error: Client connection negotiation failed: client connection to 127.20.1.189:38329: connect: Connection refused (error 111)
I20260620 01:59:44.517004 25222 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } attempt: 4
W20260620 01:59:44.520058 25222 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:38329: Network error: Client connection negotiation failed: client connection to 127.20.1.189:38329: connect: Connection refused (error 111)
W20260620 01:59:44.605914 25224 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:44.606329 25224 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:44.606422 25224 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:44.617831 25224 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:59:44.618000 25224 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:44.618068 25224 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:59:44.618119 25224 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:59:44.631374 25224 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:38329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:44.633970 25224 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:44.637574 25224 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:44.651933 25233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:44.651929 25232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:44.653141 25235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:44.655076 25224 server_base.cc:1061] running on GCE node
I20260620 01:59:44.656342 25224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:44.659257 25224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:44.662948 25224 hybrid_clock.cc:648] HybridClock initialized: now 1781920784662577 us; error 152 us; skew 500 ppm
I20260620 01:59:44.663650 25224 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:44.667946 25224 webserver.cc:533] Webserver started at http://127.20.1.189:45667/ using document root <none> and password file <none>
I20260620 01:59:44.668912 25224 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:44.669068 25224 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:44.669561 25224 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:59:44.672986 25224 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "1d15ab75054b45efa808cb1ce130243e"
format_stamp: "Formatted at 2026-06-20 01:59:44 on dist-test-slave-wg14"
I20260620 01:59:44.674877 25224 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "1d15ab75054b45efa808cb1ce130243e"
format_stamp: "Formatted at 2026-06-20 01:59:44 on dist-test-slave-wg14"
I20260620 01:59:44.685068 25224 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.003s
I20260620 01:59:44.690959 25241 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:44.695709 25224 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.005s sys 0.000s
I20260620 01:59:44.696290 25224 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "1d15ab75054b45efa808cb1ce130243e"
format_stamp: "Formatted at 2026-06-20 01:59:44 on dist-test-slave-wg14"
I20260620 01:59:44.697162 25224 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:44.725775 25224 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:44.726949 25224 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:44.727316 25224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:44.764477 25224 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:38329
I20260620 01:59:44.764434 25292 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:38329 every 8 connection(s)
I20260620 01:59:44.766604 25224 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:59:44.773370 25293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:59:44.776536 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 25224
I20260620 01:59:44.776751 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:59:44.777915 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:38969
--webserver_interface=127.20.1.188
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969 with env {}
I20260620 01:59:44.781255 25293 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:44.802663 25293 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:44.817793 25293 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:44.819455 25222 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } attempt: 5
W20260620 01:59:44.819658 25244 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:38969: connect: Connection refused (error 111)
W20260620 01:59:44.821928 25293 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:38969: Network error: Client connection negotiation failed: client connection to 127.20.1.188:38969: connect: Connection refused (error 111)
I20260620 01:59:44.832469 25222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:59:44.835433 25222 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:38969: Network error: Client connection negotiation failed: client connection to 127.20.1.188:38969: connect: Connection refused (error 111)
I20260620 01:59:44.871508 25293 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } attempt: 1
W20260620 01:59:44.875624 25293 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:38969: Network error: Client connection negotiation failed: client connection to 127.20.1.188:38969: connect: Connection refused (error 111)
I20260620 01:59:44.886831 25222 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } attempt: 1
W20260620 01:59:44.889954 25222 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:38969: Network error: Client connection negotiation failed: client connection to 127.20.1.188:38969: connect: Connection refused (error 111)
I20260620 01:59:44.944116 25293 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } attempt: 2
W20260620 01:59:44.947206 25293 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:38969: Network error: Client connection negotiation failed: client connection to 127.20.1.188:38969: connect: Connection refused (error 111)
I20260620 01:59:44.958340 25222 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } attempt: 2
W20260620 01:59:44.960886 25222 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:38969: Network error: Client connection negotiation failed: client connection to 127.20.1.188:38969: connect: Connection refused (error 111)
I20260620 01:59:45.038600 25293 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } attempt: 3
W20260620 01:59:45.043455 25293 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:38969: Network error: Client connection negotiation failed: client connection to 127.20.1.188:38969: connect: Connection refused (error 111)
I20260620 01:59:45.067255 25222 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } attempt: 3
W20260620 01:59:45.071117 25222 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:38969: Network error: Client connection negotiation failed: client connection to 127.20.1.188:38969: connect: Connection refused (error 111)
I20260620 01:59:45.186842 25293 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } attempt: 4
W20260620 01:59:45.190516 25293 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:38969: Network error: Client connection negotiation failed: client connection to 127.20.1.188:38969: connect: Connection refused (error 111)
I20260620 01:59:45.248538 25222 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } attempt: 4
W20260620 01:59:45.253202 25222 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:38969: Network error: Client connection negotiation failed: client connection to 127.20.1.188:38969: connect: Connection refused (error 111)
W20260620 01:59:45.277614 25297 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:45.278101 25297 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:45.278229 25297 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:45.289925 25297 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:59:45.290076 25297 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:45.290139 25297 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:59:45.290180 25297 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:59:45.304486 25297 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:38969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:45.307075 25297 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:45.309551 25297 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:45.319958 25305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:45.319934 25304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:45.321897 25307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:45.322494 25297 server_base.cc:1061] running on GCE node
I20260620 01:59:45.323874 25297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:45.326298 25297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:45.330180 25297 hybrid_clock.cc:648] HybridClock initialized: now 1781920785329900 us; error 130 us; skew 500 ppm
I20260620 01:59:45.331193 25297 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:45.335525 25297 webserver.cc:533] Webserver started at http://127.20.1.188:38711/ using document root <none> and password file <none>
I20260620 01:59:45.336920 25297 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:45.337071 25297 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:45.337543 25297 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:59:45.341310 25297 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "d1831cd280a541c194b3f9ee40c1d51b"
format_stamp: "Formatted at 2026-06-20 01:59:45 on dist-test-slave-wg14"
I20260620 01:59:45.342465 25297 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "d1831cd280a541c194b3f9ee40c1d51b"
format_stamp: "Formatted at 2026-06-20 01:59:45 on dist-test-slave-wg14"
I20260620 01:59:45.351570 25297 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.011s sys 0.000s
I20260620 01:59:45.357596 25313 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:45.360453 25297 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:59:45.360749 25297 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "d1831cd280a541c194b3f9ee40c1d51b"
format_stamp: "Formatted at 2026-06-20 01:59:45 on dist-test-slave-wg14"
I20260620 01:59:45.361073 25297 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:45.402132 25297 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:45.403502 25297 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:45.403855 25297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:45.436165 25297 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:38969
I20260620 01:59:45.436172 25364 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:38969 every 8 connection(s)
I20260620 01:59:45.438156 25297 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:59:45.441495 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 25297
I20260620 01:59:45.441702 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:59:45.445466 25365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:59:45.458387 25365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:45.484057 25365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:45.489866 25293 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } attempt: 5
I20260620 01:59:45.496094 25365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:45.506925 25293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e: Bootstrap starting.
I20260620 01:59:45.514405 25293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e: Neither blocks nor log segments found. Creating new log.
I20260620 01:59:45.515864 25365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b: Bootstrap starting.
I20260620 01:59:45.516208 25293 log.cc:826] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e: Log is configured to *not* fsync() on all Append() calls
I20260620 01:59:45.521435 25293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e: No bootstrap required, opened a new log
I20260620 01:59:45.521463 25365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b: Neither blocks nor log segments found. Creating new log.
I20260620 01:59:45.523156 25365 log.cc:826] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b: Log is configured to *not* fsync() on all Append() calls
I20260620 01:59:45.530023 25365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b: No bootstrap required, opened a new log
I20260620 01:59:45.530467 25222 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } attempt: 5
I20260620 01:59:45.532034 25293 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } }
I20260620 01:59:45.533334 25293 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:59:45.533586 25293 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d15ab75054b45efa808cb1ce130243e, State: Initialized, Role: FOLLOWER
I20260620 01:59:45.534837 25293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } }
I20260620 01:59:45.541780 25365 raft_consensus.cc:359] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } }
I20260620 01:59:45.542367 25365 raft_consensus.cc:385] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:59:45.542557 25365 raft_consensus.cc:740] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1831cd280a541c194b3f9ee40c1d51b, State: Initialized, Role: FOLLOWER
I20260620 01:59:45.542486 25376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } }
I20260620 01:59:45.542886 25376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:45.543582 25365 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } }
I20260620 01:59:45.545514 25293 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:59:45.546329 25222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2: Bootstrap starting.
I20260620 01:59:45.551105 25222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2: Neither blocks nor log segments found. Creating new log.
I20260620 01:59:45.553041 25222 log.cc:826] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2: Log is configured to *not* fsync() on all Append() calls
I20260620 01:59:45.555534 25377 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } }
I20260620 01:59:45.556111 25377 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:45.556949 25365 sys_catalog.cc:565] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:59:45.557025 25222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2: No bootstrap required, opened a new log
I20260620 01:59:45.565467 25222 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } }
I20260620 01:59:45.566123 25222 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:59:45.566329 25222 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f88aeb256ed4b37aa42049861c6bbc2, State: Initialized, Role: FOLLOWER
I20260620 01:59:45.567167 25222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } }
I20260620 01:59:45.584082 25395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } }
I20260620 01:59:45.584522 25395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:45.586714 25222 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:59:45.590574 25176 tablet.cc:2406] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:59:45.583701 25224 master_runner.cc:187] Error getting master registration for 127.20.1.190:42763: OK, instance_id { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" instance_seqno: 1781920784103490 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:59:45.594725 25224 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:59:45.598955 25396 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:59:45.600234 25396 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:59:45.598902 25297 master_runner.cc:187] Error getting master registration for 127.20.1.190:42763: OK, instance_id { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" instance_seqno: 1781920784103490 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:59:45.613399 25376 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:59:45.613756 25376 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } }
I20260620 01:59:45.614300 25297 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 01:59:45.616005 25376 leader_election.cc:290] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f88aeb256ed4b37aa42049861c6bbc2 (127.20.1.190:42763), d1831cd280a541c194b3f9ee40c1d51b (127.20.1.188:38969)
I20260620 01:59:45.617518 25197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d15ab75054b45efa808cb1ce130243e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" is_pre_election: true
I20260620 01:59:45.618378 25197 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d15ab75054b45efa808cb1ce130243e in term 0.
I20260620 01:59:45.619568 25242 leader_election.cc:304] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d15ab75054b45efa808cb1ce130243e, 1f88aeb256ed4b37aa42049861c6bbc2; no voters:
W20260620 01:59:45.620564 25403 catalog_manager.cc:1593] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:59:45.620764 25403 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:59:45.621160 25340 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d15ab75054b45efa808cb1ce130243e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1831cd280a541c194b3f9ee40c1d51b" is_pre_election: true
I20260620 01:59:45.621982 25340 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d15ab75054b45efa808cb1ce130243e in term 0.
I20260620 01:59:45.624169 25376 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:59:45.624395 25376 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:59:45.624501 25376 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:59:45.629479 25376 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } }
I20260620 01:59:45.631572 25197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d15ab75054b45efa808cb1ce130243e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f88aeb256ed4b37aa42049861c6bbc2"
I20260620 01:59:45.632016 25197 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:59:45.639122 25197 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d15ab75054b45efa808cb1ce130243e in term 1.
I20260620 01:59:45.640014 25376 leader_election.cc:290] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [CANDIDATE]: Term 1 election: Requested vote from peers 1f88aeb256ed4b37aa42049861c6bbc2 (127.20.1.190:42763), d1831cd280a541c194b3f9ee40c1d51b (127.20.1.188:38969)
I20260620 01:59:45.640928 25242 leader_election.cc:304] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d15ab75054b45efa808cb1ce130243e, 1f88aeb256ed4b37aa42049861c6bbc2; no voters:
I20260620 01:59:45.641155 25340 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d15ab75054b45efa808cb1ce130243e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1831cd280a541c194b3f9ee40c1d51b"
I20260620 01:59:45.641467 25340 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:59:45.643784 25412 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:59:45.644083 25412 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [term 1 LEADER]: Becoming Leader. State: Replica: 1d15ab75054b45efa808cb1ce130243e, State: Running, Role: LEADER
I20260620 01:59:45.644770 25412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } }
I20260620 01:59:45.645615 25340 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d15ab75054b45efa808cb1ce130243e in term 1.
W20260620 01:59:45.657053 25319 tablet.cc:2406] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:59:45.660562 25376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d15ab75054b45efa808cb1ce130243e. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } }
I20260620 01:59:45.660949 25376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:45.675366 25418 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:59:45.680485 25418 catalog_manager.cc:1519] Initializing Kudu cluster ID...
W20260620 01:59:45.683305 25415 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:59:45.683704 25415 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:59:45.693315 25340 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [term 1 FOLLOWER]: Refusing update from remote peer 1d15ab75054b45efa808cb1ce130243e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:59:45.695242 25412 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:59:45.698722 25197 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [term 1 FOLLOWER]: Refusing update from remote peer 1d15ab75054b45efa808cb1ce130243e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:59:45.703171 25412 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:59:45.727325 25377 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d15ab75054b45efa808cb1ce130243e. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } }
I20260620 01:59:45.727692 25377 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:45.727809 25419 mvcc.cc:204] Tried to move back new op lower bound from 7298747538178682880 to 7298747538024083456. Current Snapshot: MvccSnapshot[applied={T|T < 7298747538178682880}]
I20260620 01:59:45.735620 25420 mvcc.cc:204] Tried to move back new op lower bound from 7298747538178682880 to 7298747538024083456. Current Snapshot: MvccSnapshot[applied={T|T < 7298747538178682880}]
I20260620 01:59:45.742892 25412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } }
I20260620 01:59:45.743287 25412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:45.746687 25418 catalog_manager.cc:1382] Generated new cluster ID: 465359e0241b4406b8eda8328b6d5625
I20260620 01:59:45.747574 25418 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 01:59:45.753782 25395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d15ab75054b45efa808cb1ce130243e. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } }
I20260620 01:59:45.754209 25395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:45.760442 25377 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } }
I20260620 01:59:45.760689 25377 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:45.769220 25376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } }
I20260620 01:59:45.769608 25376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:45.781977 25395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } }
I20260620 01:59:45.782281 25395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:45.821062 25418 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:59:45.828526 25418 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:59:45.880440 25418 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e: Generated new TSK 0
I20260620 01:59:45.881968 25418 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:59:45.916282 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260620 01:59:46.623646 25403 catalog_manager.cc:1294] Loaded cluster ID: 465359e0241b4406b8eda8328b6d5625
I20260620 01:59:46.623795 25403 catalog_manager.cc:1587] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b: loading cluster ID for follower catalog manager: success
I20260620 01:59:46.629194 25403 catalog_manager.cc:1609] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b: acquiring CA information for follower catalog manager: success
I20260620 01:59:46.632490 25403 catalog_manager.cc:1637] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 01:59:46.646088 25430 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:46.646479 25430 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:46.646687 25430 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:46.657778 25430 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:46.657997 25430 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:59:46.671391 25430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:46.674048 25430 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:46.676544 25430 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260620 01:59:46.687548 25415 catalog_manager.cc:1294] Loaded cluster ID: 465359e0241b4406b8eda8328b6d5625
I20260620 01:59:46.687948 25415 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2: loading cluster ID for follower catalog manager: success
W20260620 01:59:46.690726 25435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:46.690938 25436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:46.693452 25415 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2: acquiring CA information for follower catalog manager: success
W20260620 01:59:46.693531 25438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:46.693820 25430 server_base.cc:1061] running on GCE node
I20260620 01:59:46.694769 25430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:46.696367 25430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:46.696375 25415 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:59:46.697937 25430 hybrid_clock.cc:648] HybridClock initialized: now 1781920786697757 us; error 122 us; skew 500 ppm
I20260620 01:59:46.698491 25430 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:46.702452 25430 webserver.cc:533] Webserver started at http://127.20.1.129:37029/ using document root <none> and password file <none>
I20260620 01:59:46.703675 25430 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:46.703864 25430 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:46.704286 25430 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:59:46.707569 25430 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "f6a844e9c6d8422ba9cf5faca622048f"
format_stamp: "Formatted at 2026-06-20 01:59:46 on dist-test-slave-wg14"
I20260620 01:59:46.708773 25430 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "f6a844e9c6d8422ba9cf5faca622048f"
format_stamp: "Formatted at 2026-06-20 01:59:46 on dist-test-slave-wg14"
I20260620 01:59:46.716814 25430 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260620 01:59:46.722784 25444 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:46.725028 25430 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260620 01:59:46.725302 25430 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "f6a844e9c6d8422ba9cf5faca622048f"
format_stamp: "Formatted at 2026-06-20 01:59:46 on dist-test-slave-wg14"
I20260620 01:59:46.725596 25430 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:46.754503 25430 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:46.755702 25430 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:46.756006 25430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:46.757210 25430 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:59:46.759704 25430 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:59:46.759856 25430 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:46.759990 25430 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:59:46.760066 25430 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:46.810763 25430 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:40011
I20260620 01:59:46.810855 25556 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:40011 every 8 connection(s)
I20260620 01:59:46.812714 25430 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:59:46.815035 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 25430
I20260620 01:59:46.815440 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 01:59:46.853745 25557 heartbeater.cc:344] Connected to a master server at 127.20.1.188:38969
I20260620 01:59:46.854290 25558 heartbeater.cc:344] Connected to a master server at 127.20.1.190:42763
I20260620 01:59:46.854600 25558 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:46.854686 25557 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:46.856827 25557 heartbeater.cc:507] Master 127.20.1.188:38969 requested a full tablet report, sending...
I20260620 01:59:46.861418 25558 heartbeater.cc:507] Master 127.20.1.190:42763 requested a full tablet report, sending...
I20260620 01:59:46.861835 25330 ts_manager.cc:194] Registered new tserver with Master: f6a844e9c6d8422ba9cf5faca622048f (127.20.1.129:40011)
I20260620 01:59:46.863930 25187 ts_manager.cc:194] Registered new tserver with Master: f6a844e9c6d8422ba9cf5faca622048f (127.20.1.129:40011)
I20260620 01:59:46.866642 25559 heartbeater.cc:344] Connected to a master server at 127.20.1.189:38329
I20260620 01:59:46.866971 25559 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:46.867805 25559 heartbeater.cc:507] Master 127.20.1.189:38329 requested a full tablet report, sending...
I20260620 01:59:46.870167 25258 ts_manager.cc:194] Registered new tserver with Master: f6a844e9c6d8422ba9cf5faca622048f (127.20.1.129:40011)
I20260620 01:59:46.872076 25258 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:55001
I20260620 01:59:46.889721 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:59:46.893227 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:41499
--webserver_interface=127.20.1.187
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969,127.20.1.187:41499 with env {}
W20260620 01:59:47.354564 25567 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:47.354969 25567 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:47.355068 25567 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:47.364769 25567 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:59:47.364950 25567 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:47.365017 25567 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:59:47.365067 25567 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:59:47.378280 25567 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969,127.20.1.187:41499
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:41499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.20.1.187
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:47.380582 25567 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:47.384541 25567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:47.395257 25573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:47.395762 25572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:47.396525 25575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:47.397253 25567 server_base.cc:1061] running on GCE node
I20260620 01:59:47.398324 25567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:47.400682 25567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:47.404229 25567 hybrid_clock.cc:648] HybridClock initialized: now 1781920787404015 us; error 117 us; skew 500 ppm
I20260620 01:59:47.404917 25567 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:47.409443 25567 webserver.cc:533] Webserver started at http://127.20.1.187:41409/ using document root <none> and password file <none>
I20260620 01:59:47.410423 25567 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:47.410555 25567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:47.410930 25567 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:59:47.414458 25567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/data/instance:
uuid: "3ab983ee45b74499968ae51f953f0204"
format_stamp: "Formatted at 2026-06-20 01:59:47 on dist-test-slave-wg14"
I20260620 01:59:47.416085 25567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/wal/instance:
uuid: "3ab983ee45b74499968ae51f953f0204"
format_stamp: "Formatted at 2026-06-20 01:59:47 on dist-test-slave-wg14"
I20260620 01:59:47.424386 25567 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s
I20260620 01:59:47.429499 25581 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:47.432847 25567 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 01:59:47.433229 25567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "3ab983ee45b74499968ae51f953f0204"
format_stamp: "Formatted at 2026-06-20 01:59:47 on dist-test-slave-wg14"
I20260620 01:59:47.433593 25567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:47.473272 25567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:47.474777 25567 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:47.475198 25567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:47.503407 25632 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:41499 every 8 connection(s)
I20260620 01:59:47.503407 25567 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:41499
I20260620 01:59:47.505988 25567 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 01:59:47.508764 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 25567
I20260620 01:59:47.513590 25633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:59:47.523545 25633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:47.537432 25633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:47.546891 25633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:47.556121 25633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } has no permanent_uuid. Determining permanent_uuid...
I20260620 01:59:47.570454 25633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: Bootstrap starting.
I20260620 01:59:47.574699 25633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: Neither blocks nor log segments found. Creating new log.
I20260620 01:59:47.576110 25633 log.cc:826] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: Log is configured to *not* fsync() on all Append() calls
I20260620 01:59:47.581007 25633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: No bootstrap required, opened a new log
I20260620 01:59:47.588562 25633 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } }
I20260620 01:59:47.589099 25633 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:59:47.589226 25633 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab983ee45b74499968ae51f953f0204, State: Initialized, Role: FOLLOWER
I20260620 01:59:47.590289 25633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } }
I20260620 01:59:47.591645 25642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } } }
I20260620 01:59:47.592054 25642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:47.592896 25633 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:59:47.606361 25653 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:59:47.606570 25653 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:59:47.618582 25642 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:59:47.618949 25567 master_runner.cc:429] Detected that this master 3ab983ee45b74499968ae51f953f0204 is joining an existing cluster
I20260620 01:59:47.618940 25642 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } }
I20260620 01:59:47.619439 25567 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260620 01:59:47.620862 25567 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 01:59:47.621486 25567 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 01:59:47.621764 25642 leader_election.cc:290] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f88aeb256ed4b37aa42049861c6bbc2 (127.20.1.190:42763), 1d15ab75054b45efa808cb1ce130243e (127.20.1.189:38329), d1831cd280a541c194b3f9ee40c1d51b (127.20.1.188:38969)
I20260620 01:59:47.622465 25567 tablet_replica.cc:333] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: stopping tablet replica
I20260620 01:59:47.622917 25339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ab983ee45b74499968ae51f953f0204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1831cd280a541c194b3f9ee40c1d51b" is_pre_election: true
I20260620 01:59:47.623086 25268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ab983ee45b74499968ae51f953f0204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d15ab75054b45efa808cb1ce130243e" is_pre_election: true
I20260620 01:59:47.624280 25197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ab983ee45b74499968ae51f953f0204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" is_pre_election: true
I20260620 01:59:47.625211 25567 master_runner.cc:434] Initiating AddMaster RPC to add 127.20.1.187:41499
I20260620 01:59:47.625357 25582 leader_election.cc:304] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3ab983ee45b74499968ae51f953f0204; no voters: 1f88aeb256ed4b37aa42049861c6bbc2, d1831cd280a541c194b3f9ee40c1d51b
W20260620 01:59:47.625823 25582 raft_consensus.cc:2686] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260620 01:59:47.684689 25258 catalog_manager.cc:7201] Initiating ChangeConfig request to add master 127.20.1.187:41499: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3ab983ee45b74499968ae51f953f0204"
member_type: NON_VOTER
last_known_addr {
host: "127.20.1.187"
port: 41499
}
attrs {
promote: true
}
}
dest_uuid: "1d15ab75054b45efa808cb1ce130243e"
cas_config_opid_index: -1
I20260620 01:59:47.687058 25258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: true } }
I20260620 01:59:47.692533 25197 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [term 1 FOLLOWER]: Refusing update from remote peer 1d15ab75054b45efa808cb1ce130243e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 01:59:47.694021 25428 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260620 01:59:47.694638 25245 consensus_peers.cc:597] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e -> Peer 3ab983ee45b74499968ae51f953f0204 (127.20.1.187:41499): Couldn't send request to peer 3ab983ee45b74499968ae51f953f0204. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260620 01:59:47.697438 25339 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [term 1 FOLLOWER]: Refusing update from remote peer 1d15ab75054b45efa808cb1ce130243e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 01:59:47.699445 25665 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 01:59:47.698696 25428 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3ab983ee45b74499968ae51f953f0204 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: true } } }
I20260620 01:59:47.699703 25197 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3ab983ee45b74499968ae51f953f0204 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: true } } }
I20260620 01:59:47.703395 25428 catalog_manager.cc:7205] Successfully completed master ChangeConfig request to add master 127.20.1.187:41499
I20260620 01:59:47.704521 25567 master.cc:562] Master@127.20.1.187:41499 shutting down...
I20260620 01:59:47.717202 25665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: true } } }
I20260620 01:59:47.717715 25665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:47.721830 25428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: true } } }
I20260620 01:59:47.721732 25669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: true } } }
I20260620 01:59:47.722190 25669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:47.722863 25428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:47.723315 25339 raft_consensus.cc:2955] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3ab983ee45b74499968ae51f953f0204 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: true } } }
I20260620 01:59:47.730715 25567 master.cc:584] Master@127.20.1.187:41499 shutdown complete.
I20260620 01:59:47.731029 25567 master_runner.cc:306] Clearing existing system tablet
I20260620 01:59:47.730912 25671 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: true } } }
I20260620 01:59:47.731616 25567 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:47.731581 25671 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:47.731832 25567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:47.735440 25567 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 01:59:47.737888 25673 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:47.738670 25567 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:47.739176 25567 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 01:59:47.739351 25567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "3ab983ee45b74499968ae51f953f0204"
format_stamp: "Formatted at 2026-06-20 01:59:47 on dist-test-slave-wg14"
I20260620 01:59:47.745086 25567 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 01:59:47.753697 25567 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 01:59:47.754046 25567 log.cc:1199] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: Deleting WAL directory at /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260620 01:59:47.754652 25567 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: Deleting consensus metadata
I20260620 01:59:47.754966 25567 master_runner.cc:316] Copying system tablet from 127.20.1.189:38329
I20260620 01:59:47.758399 25567 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: tablet copy: Beginning tablet copy session from remote peer at address 127.20.1.189:38329
I20260620 01:59:47.769919 25278 tablet_copy_service.cc:140] P 1d15ab75054b45efa808cb1ce130243e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3ab983ee45b74499968ae51f953f0204 ({username='slave'} at 127.0.0.1:39650)
I20260620 01:59:47.770308 25278 tablet_copy_service.cc:161] P 1d15ab75054b45efa808cb1ce130243e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3ab983ee45b74499968ae51f953f0204 at {username='slave'} at 127.0.0.1:39650: session id = 3ab983ee45b74499968ae51f953f0204-00000000000000000000000000000000
I20260620 01:59:47.776444 25278 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 01:59:47.779904 25567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 01:59:47.789022 25567 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: tablet copy: Starting download of 0 data blocks...
I20260620 01:59:47.789433 25567 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: tablet copy: Starting download of 1 WAL segments...
I20260620 01:59:47.793570 25567 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 01:59:47.799113 25278 tablet_copy_service.cc:342] P 1d15ab75054b45efa808cb1ce130243e: Request end of tablet copy session 3ab983ee45b74499968ae51f953f0204-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:39650
I20260620 01:59:47.799422 25278 tablet_copy_service.cc:434] P 1d15ab75054b45efa808cb1ce130243e: ending tablet copy session 3ab983ee45b74499968ae51f953f0204-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3ab983ee45b74499968ae51f953f0204
W20260620 01:59:47.808210 25567 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260620 01:59:47.813431 25567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:47.816906 25680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:47.817751 25681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:47.819424 25567 server_base.cc:1061] running on GCE node
W20260620 01:59:47.820523 25683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:47.821079 25567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:47.822230 25567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:47.823791 25567 hybrid_clock.cc:648] HybridClock initialized: now 1781920787823588 us; error 117 us; skew 500 ppm
I20260620 01:59:47.824141 25567 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:47.826383 25567 webserver.cc:533] Webserver started at http://127.20.1.187:42513/ using document root <none> and password file <none>
I20260620 01:59:47.826882 25567 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:47.827005 25567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:47.830531 25567 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 01:59:47.833370 25689 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:47.833985 25567 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 01:59:47.834182 25567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "3ab983ee45b74499968ae51f953f0204"
format_stamp: "Formatted at 2026-06-20 01:59:47 on dist-test-slave-wg14"
I20260620 01:59:47.834440 25567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:47.847681 25567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:47.848734 25567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:47.874970 25567 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:41499
I20260620 01:59:47.875000 25740 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:41499 every 8 connection(s)
I20260620 01:59:47.876410 25567 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 01:59:47.876583 25559 heartbeater.cc:499] Master 127.20.1.189:38329 was elected leader, sending a full tablet report...
I20260620 01:59:47.881410 25741 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:59:47.884591 25741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: Bootstrap starting.
I20260620 01:59:47.905900 25741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:59:47.906550 25741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: Bootstrap complete.
I20260620 01:59:47.908362 25741 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: true } }
I20260620 01:59:47.908741 25741 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3ab983ee45b74499968ae51f953f0204, State: Initialized, Role: LEARNER
I20260620 01:59:47.909090 25741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: true } }
I20260620 01:59:47.910458 25745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: true } } }
I20260620 01:59:47.910969 25741 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:59:47.911010 25745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [sys.catalog]: This master's current role is: LEARNER
I20260620 01:59:47.926894 25756 catalog_manager.cc:1294] Loaded cluster ID: 465359e0241b4406b8eda8328b6d5625
I20260620 01:59:47.927109 25756 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: loading cluster ID for follower catalog manager: success
I20260620 01:59:47.930835 25756 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: acquiring CA information for follower catalog manager: success
I20260620 01:59:47.933907 25756 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:59:47.957931 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 25430
I20260620 01:59:47.975960 25552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:48.075618 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 25430
I20260620 01:59:48.100558 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:40011
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969
--webserver_port=37029
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969,127.20.1.187:41499 with env {}
I20260620 01:59:48.164971 25668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: true } } }
I20260620 01:59:48.165489 25668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:48.303094 25716 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260620 01:59:48.304111 25745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d15ab75054b45efa808cb1ce130243e. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: true } } }
I20260620 01:59:48.304489 25745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [sys.catalog]: This master's current role is: LEARNER
W20260620 01:59:48.574846 25759 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:48.575248 25759 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:48.575387 25759 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:48.585456 25759 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:48.585767 25759 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:59:48.598994 25759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:40011
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=37029
--tserver_master_addrs=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969,127.20.1.187:41499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:48.600945 25759 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:48.603116 25759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:48.615501 25765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:48.616744 25766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:48.617007 25768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:48.619246 25759 server_base.cc:1061] running on GCE node
I20260620 01:59:48.620116 25759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:48.621922 25759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:48.625613 25759 hybrid_clock.cc:648] HybridClock initialized: now 1781920788625313 us; error 142 us; skew 500 ppm
I20260620 01:59:48.626262 25759 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:48.629858 25759 webserver.cc:533] Webserver started at http://127.20.1.129:37029/ using document root <none> and password file <none>
I20260620 01:59:48.631335 25759 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:48.631554 25759 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:48.639914 25759 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 01:59:48.644613 25774 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:48.646775 25759 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:59:48.647085 25759 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "f6a844e9c6d8422ba9cf5faca622048f"
format_stamp: "Formatted at 2026-06-20 01:59:46 on dist-test-slave-wg14"
I20260620 01:59:48.648211 25759 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:48.665803 25759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:48.667656 25759 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:48.669080 25759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:48.670809 25759 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:59:48.673976 25759 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:59:48.674119 25759 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:48.674240 25759 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:59:48.674289 25759 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:48.722334 25759 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:40011
I20260620 01:59:48.722437 25886 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:40011 every 8 connection(s)
I20260620 01:59:48.724730 25759 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:59:48.729391 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 25759
I20260620 01:59:48.761439 25888 heartbeater.cc:344] Connected to a master server at 127.20.1.188:38969
I20260620 01:59:48.761922 25888 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:48.764590 25887 heartbeater.cc:344] Connected to a master server at 127.20.1.187:41499
I20260620 01:59:48.765013 25887 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:48.765515 25428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: true } } }
I20260620 01:59:48.765853 25887 heartbeater.cc:507] Master 127.20.1.187:41499 requested a full tablet report, sending...
I20260620 01:59:48.765944 25428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:48.769937 25706 ts_manager.cc:194] Registered new tserver with Master: f6a844e9c6d8422ba9cf5faca622048f (127.20.1.129:40011)
I20260620 01:59:48.764369 25888 heartbeater.cc:507] Master 127.20.1.188:38969 requested a full tablet report, sending...
I20260620 01:59:48.776021 25330 ts_manager.cc:194] Re-registered known tserver with Master: f6a844e9c6d8422ba9cf5faca622048f (127.20.1.129:40011)
I20260620 01:59:48.777647 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:59:48.778527 25899 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e: attempting to promote NON_VOTER 3ab983ee45b74499968ae51f953f0204 to VOTER
I20260620 01:59:48.778574 25890 heartbeater.cc:344] Connected to a master server at 127.20.1.189:38329
I20260620 01:59:48.778772 25890 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:48.779649 25890 heartbeater.cc:507] Master 127.20.1.189:38329 requested a full tablet report, sending...
I20260620 01:59:48.780088 25899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } }
I20260620 01:59:48.786366 25258 ts_manager.cc:194] Re-registered known tserver with Master: f6a844e9c6d8422ba9cf5faca622048f (127.20.1.129:40011)
I20260620 01:59:48.787066 25339 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [term 1 FOLLOWER]: Refusing update from remote peer 1d15ab75054b45efa808cb1ce130243e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 01:59:48.788178 25258 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:36243
I20260620 01:59:48.791253 25197 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [term 1 FOLLOWER]: Refusing update from remote peer 1d15ab75054b45efa808cb1ce130243e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 01:59:48.791239 25428 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:59:48.794745 25716 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [term 1 LEARNER]: Refusing update from remote peer 1d15ab75054b45efa808cb1ce130243e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 01:59:48.795308 25903 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:59:48.796013 25889 heartbeater.cc:344] Connected to a master server at 127.20.1.190:42763
I20260620 01:59:48.796640 25889 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:48.797247 25889 heartbeater.cc:507] Master 127.20.1.190:42763 requested a full tablet report, sending...
I20260620 01:59:48.797197 25903 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 01:59:48.805056 25903 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3ab983ee45b74499968ae51f953f0204 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } } }
I20260620 01:59:48.806099 25187 ts_manager.cc:194] Re-registered known tserver with Master: f6a844e9c6d8422ba9cf5faca622048f (127.20.1.129:40011)
I20260620 01:59:48.807428 25339 raft_consensus.cc:2955] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3ab983ee45b74499968ae51f953f0204 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } } }
I20260620 01:59:48.808955 25197 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3ab983ee45b74499968ae51f953f0204 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } } }
I20260620 01:59:48.810842 25428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } } }
I20260620 01:59:48.811211 25428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:48.811093 25899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } } }
I20260620 01:59:48.811405 25899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:48.811652 25899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } } }
I20260620 01:59:48.811627 25428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } } }
I20260620 01:59:48.811900 25428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:48.812083 25899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:48.812961 25716 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3ab983ee45b74499968ae51f953f0204 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } } }
I20260620 01:59:48.818439 25908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } } }
I20260620 01:59:48.819072 25908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:48.819116 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:59:48.819511 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 25156
I20260620 01:59:48.821527 25217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:48.824311 25907 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } } }
I20260620 01:59:48.825034 25907 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1831cd280a541c194b3f9ee40c1d51b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:48.825654 25909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } } }
I20260620 01:59:48.826138 25909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ab983ee45b74499968ae51f953f0204 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:48.970908 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 25156
I20260620 01:59:48.992071 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42763
--webserver_interface=127.20.1.190
--webserver_port=34373
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969
--master_addresses=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969,127.20.1.187:41499 with env {}
W20260620 01:59:49.290699 25242 consensus_peers.cc:597] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e -> Peer 1f88aeb256ed4b37aa42049861c6bbc2 (127.20.1.190:42763): Couldn't send request to peer 1f88aeb256ed4b37aa42049861c6bbc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:42763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260620 01:59:49.455860 25911 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:49.456341 25911 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:49.456461 25911 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:49.469231 25911 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:59:49.469408 25911 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:49.469482 25911 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:59:49.469574 25911 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:59:49.483588 25911 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969,127.20.1.187:41499
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=34373
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:49.486135 25911 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:49.488497 25911 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:49.501791 25918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:49.501791 25919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:49.502781 25921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:49.503670 25911 server_base.cc:1061] running on GCE node
I20260620 01:59:49.504805 25911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:49.507783 25911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:49.511376 25911 hybrid_clock.cc:648] HybridClock initialized: now 1781920789511120 us; error 157 us; skew 500 ppm
I20260620 01:59:49.512012 25911 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:49.516207 25911 webserver.cc:533] Webserver started at http://127.20.1.190:34373/ using document root <none> and password file <none>
I20260620 01:59:49.517460 25911 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:49.517822 25911 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:49.527108 25911 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 01:59:49.531606 25927 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:49.534005 25911 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 01:59:49.534305 25911 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "1f88aeb256ed4b37aa42049861c6bbc2"
format_stamp: "Formatted at 2026-06-20 01:59:44 on dist-test-slave-wg14"
I20260620 01:59:49.536211 25911 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:49.565356 25911 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:49.566548 25911 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:49.566983 25911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:49.597169 25911 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:42763
I20260620 01:59:49.597178 25978 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:42763 every 8 connection(s)
I20260620 01:59:49.599988 25911 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:59:49.601058 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 25911
I20260620 01:59:49.601492 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 25759
I20260620 01:59:49.616405 25979 sys_catalog.cc:263] Verifying existing consensus state
I20260620 01:59:49.623636 25979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2: Bootstrap starting.
I20260620 01:59:49.631546 25882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:49.648049 25979 log.cc:826] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2: Log is configured to *not* fsync() on all Append() calls
I20260620 01:59:49.662848 25979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 01:59:49.663823 25979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2: Bootstrap complete.
I20260620 01:59:49.672681 25979 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } }
I20260620 01:59:49.673723 25979 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f88aeb256ed4b37aa42049861c6bbc2, State: Initialized, Role: FOLLOWER
I20260620 01:59:49.674865 25979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } }
I20260620 01:59:49.676370 25985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } } }
I20260620 01:59:49.677338 25985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:49.678841 25979 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:59:49.709437 25996 catalog_manager.cc:1294] Loaded cluster ID: 465359e0241b4406b8eda8328b6d5625
I20260620 01:59:49.709600 25996 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2: loading cluster ID for follower catalog manager: success
I20260620 01:59:49.719595 25996 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2: acquiring CA information for follower catalog manager: success
I20260620 01:59:49.723208 25996 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 01:59:49.836390 25899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } } }
I20260620 01:59:49.836992 25899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:49.855425 25985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d15ab75054b45efa808cb1ce130243e. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } } }
I20260620 01:59:49.855784 25985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f88aeb256ed4b37aa42049861c6bbc2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:49.861173 25890 heartbeater.cc:499] Master 127.20.1.189:38329 was elected leader, sending a full tablet report...
I20260620 01:59:49.867151 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 25759
I20260620 01:59:49.892149 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:40011
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969
--webserver_port=37029
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969,127.20.1.187:41499 with env {}
W20260620 01:59:49.892179 25998 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 01:59:49.863860 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:44121 (local address 127.20.1.190:42763)
0620 01:59:49.865754 (+ 1894us) server_negotiation.cc:207] Beginning negotiation
0620 01:59:49.865762 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 01:59:49.866348 (+ 586us) server_negotiation.cc:408] Connection header received
0620 01:59:49.866834 (+ 486us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 01:59:49.866842 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 01:59:49.866935 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 01:59:49.867005 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0620 01:59:49.890437 (+ 23432us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:44121: BlockingRecv error: recv got EOF from 127.20.1.129:44121 (error 108)
Metrics: {"server-negotiator.queue_time_us":1687}
I20260620 01:59:50.271966 25899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d15ab75054b45efa808cb1ce130243e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f88aeb256ed4b37aa42049861c6bbc2" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42763 } } peers { permanent_uuid: "1d15ab75054b45efa808cb1ce130243e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 38329 } } peers { permanent_uuid: "d1831cd280a541c194b3f9ee40c1d51b" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 38969 } } peers { permanent_uuid: "3ab983ee45b74499968ae51f953f0204" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 41499 } attrs { promote: false } } }
I20260620 01:59:50.272390 25899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d15ab75054b45efa808cb1ce130243e [sys.catalog]: This master's current role is: LEADER
W20260620 01:59:50.466142 26007 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:50.467072 26007 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:50.467785 26007 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:50.479027 26007 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:50.479394 26007 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:59:50.492908 26007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:40011
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=37029
--tserver_master_addrs=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969,127.20.1.187:41499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:50.495685 26007 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:50.498287 26007 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:50.514424 26014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:50.514549 26013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:50.515906 26016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:50.517228 26007 server_base.cc:1061] running on GCE node
I20260620 01:59:50.518134 26007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:50.520067 26007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:50.523763 26007 hybrid_clock.cc:648] HybridClock initialized: now 1781920790523505 us; error 161 us; skew 500 ppm
I20260620 01:59:50.524309 26007 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:50.527648 26007 webserver.cc:533] Webserver started at http://127.20.1.129:37029/ using document root <none> and password file <none>
I20260620 01:59:50.528709 26007 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:50.528855 26007 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:50.538040 26007 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 01:59:50.542788 26022 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:50.545087 26007 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 01:59:50.545353 26007 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "f6a844e9c6d8422ba9cf5faca622048f"
format_stamp: "Formatted at 2026-06-20 01:59:46 on dist-test-slave-wg14"
I20260620 01:59:50.546548 26007 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:50.583488 26007 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:50.584631 26007 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:50.585044 26007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:50.586493 26007 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:59:50.589143 26007 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:59:50.589269 26007 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:50.589411 26007 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:59:50.589468 26007 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:50.666913 26007 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:40011
I20260620 01:59:50.666963 26136 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:40011 every 8 connection(s)
I20260620 01:59:50.673017 26007 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:59:50.675061 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 26007
I20260620 01:59:50.709126 26139 heartbeater.cc:344] Connected to a master server at 127.20.1.190:42763
I20260620 01:59:50.709853 26139 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:50.711786 26139 heartbeater.cc:507] Master 127.20.1.190:42763 requested a full tablet report, sending...
I20260620 01:59:50.715996 25944 ts_manager.cc:194] Registered new tserver with Master: f6a844e9c6d8422ba9cf5faca622048f (127.20.1.129:40011)
I20260620 01:59:50.731334 26138 heartbeater.cc:344] Connected to a master server at 127.20.1.188:38969
I20260620 01:59:50.731779 26138 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:50.733197 26138 heartbeater.cc:507] Master 127.20.1.188:38969 requested a full tablet report, sending...
I20260620 01:59:50.734048 26137 heartbeater.cc:344] Connected to a master server at 127.20.1.187:41499
I20260620 01:59:50.734237 26137 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:50.734931 26137 heartbeater.cc:507] Master 127.20.1.187:41499 requested a full tablet report, sending...
I20260620 01:59:50.736742 26140 heartbeater.cc:344] Connected to a master server at 127.20.1.189:38329
I20260620 01:59:50.737329 25330 ts_manager.cc:194] Re-registered known tserver with Master: f6a844e9c6d8422ba9cf5faca622048f (127.20.1.129:40011)
I20260620 01:59:50.739698 25706 ts_manager.cc:194] Re-registered known tserver with Master: f6a844e9c6d8422ba9cf5faca622048f (127.20.1.129:40011)
I20260620 01:59:50.737785 26140 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:50.741587 26140 heartbeater.cc:507] Master 127.20.1.189:38329 requested a full tablet report, sending...
I20260620 01:59:50.742961 25258 ts_manager.cc:194] Re-registered known tserver with Master: f6a844e9c6d8422ba9cf5faca622048f (127.20.1.129:40011)
I20260620 01:59:50.744560 25258 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:32885
I20260620 01:59:50.784415 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 26007
I20260620 01:59:50.807348 26132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:50.911252 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 26007
I20260620 01:59:50.930469 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:40011
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969
--webserver_port=37029
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969,127.20.1.187:41499
--fault_before_start=1.0 with env {}
W20260620 01:59:51.396970 26152 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:51.397382 26152 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:51.397524 26152 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 01:59:51.397635 26152 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:51.408447 26152 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:51.408775 26152 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:59:51.421133 26152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:40011
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=37029
--tserver_master_addrs=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969,127.20.1.187:41499
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:51.423207 26152 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:51.425815 26152 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:51.441205 26157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:51.441601 26158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:51.444690 26160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:51.445772 26152 server_base.cc:1061] running on GCE node
I20260620 01:59:51.446681 26152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:51.448208 26152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:51.451761 26152 hybrid_clock.cc:648] HybridClock initialized: now 1781920791451309 us; error 347 us; skew 500 ppm
I20260620 01:59:51.452283 26152 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:51.456115 26152 webserver.cc:533] Webserver started at http://127.20.1.129:37029/ using document root <none> and password file <none>
I20260620 01:59:51.457383 26152 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:51.457587 26152 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:51.466538 26152 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 01:59:51.471652 26166 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:51.474666 26152 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s
I20260620 01:59:51.474987 26152 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "f6a844e9c6d8422ba9cf5faca622048f"
format_stamp: "Formatted at 2026-06-20 01:59:46 on dist-test-slave-wg14"
I20260620 01:59:51.476338 26152 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:51.497834 26152 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:51.499095 26152 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:51.499490 26152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:51.500763 26152 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:59:51.503224 26152 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:59:51.503379 26152 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 01:59:51.503545 26152 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:59:51.503620 26152 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 01:59:51.504166 26152 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 01:59:51.504284 26152 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 01:59:51.504366 26152 fault_injection.cc:59] injecting fault for kudu (pid 26152): FLAGS_fault_before_start (process will exit)
I20260620 01:59:51.522693 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:40011
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969
--webserver_port=37029
--webserver_interface=127.20.1.129
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969,127.20.1.187:41499 with env {}
W20260620 01:59:51.993379 26174 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:51.993988 26174 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:51.994153 26174 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:52.005656 26174 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:52.005932 26174 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:59:52.018669 26174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45445,127.20.1.149:36423,127.20.1.150:36973,127.20.1.151:46625,127.20.1.152:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:40011
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=37029
--tserver_master_addrs=127.20.1.190:42763,127.20.1.189:38329,127.20.1.188:38969,127.20.1.187:41499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:52.020632 26174 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:52.023490 26174 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:52.042011 26179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:52.042137 26182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:52.041993 26180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:52.043437 26174 server_base.cc:1061] running on GCE node
I20260620 01:59:52.044297 26174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 01:59:52.046084 26174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 01:59:52.049649 26174 hybrid_clock.cc:648] HybridClock initialized: now 1781920792049429 us; error 115 us; skew 500 ppm
I20260620 01:59:52.050158 26174 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 01:59:52.054522 26174 webserver.cc:533] Webserver started at http://127.20.1.129:37029/ using document root <none> and password file <none>
I20260620 01:59:52.055673 26174 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:52.055830 26174 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:52.064613 26174 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 01:59:52.070724 26188 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:52.073988 26174 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.001s sys 0.004s
I20260620 01:59:52.074292 26174 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "f6a844e9c6d8422ba9cf5faca622048f"
format_stamp: "Formatted at 2026-06-20 01:59:46 on dist-test-slave-wg14"
I20260620 01:59:52.075362 26174 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:52.096274 26174 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:52.097429 26174 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:52.097756 26174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:52.099674 26174 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:59:52.102735 26174 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:59:52.102867 26174 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:52.103014 26174 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:59:52.103072 26174 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:52.152380 26174 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:40011
I20260620 01:59:52.152493 26300 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:40011 every 8 connection(s)
I20260620 01:59:52.154397 26174 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:59:52.163632 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 26174
I20260620 01:59:52.164041 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 26174
I20260620 01:59:52.216351 26303 heartbeater.cc:344] Connected to a master server at 127.20.1.190:42763
I20260620 01:59:52.216905 26303 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:52.217145 26302 heartbeater.cc:344] Connected to a master server at 127.20.1.188:38969
I20260620 01:59:52.217294 26302 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:52.218454 26303 heartbeater.cc:507] Master 127.20.1.190:42763 requested a full tablet report, sending...
I20260620 01:59:52.218493 26302 heartbeater.cc:507] Master 127.20.1.188:38969 requested a full tablet report, sending...
I20260620 01:59:52.219300 26301 heartbeater.cc:344] Connected to a master server at 127.20.1.187:41499
I20260620 01:59:52.219535 26301 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:52.219869 26304 heartbeater.cc:344] Connected to a master server at 127.20.1.189:38329
I20260620 01:59:52.220072 26304 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:52.220310 26301 heartbeater.cc:507] Master 127.20.1.187:41499 requested a full tablet report, sending...
I20260620 01:59:52.221261 26304 heartbeater.cc:507] Master 127.20.1.189:38329 requested a full tablet report, sending...
I20260620 01:59:52.221346 25944 ts_manager.cc:194] Re-registered known tserver with Master: f6a844e9c6d8422ba9cf5faca622048f (127.20.1.129:40011)
I20260620 01:59:52.222586 25258 ts_manager.cc:194] Re-registered known tserver with Master: f6a844e9c6d8422ba9cf5faca622048f (127.20.1.129:40011)
I20260620 01:59:52.223757 25258 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.1.129:45533
I20260620 01:59:52.224182 25706 ts_manager.cc:194] Re-registered known tserver with Master: f6a844e9c6d8422ba9cf5faca622048f (127.20.1.129:40011)
I20260620 01:59:52.225426 25330 ts_manager.cc:194] Re-registered known tserver with Master: f6a844e9c6d8422ba9cf5faca622048f (127.20.1.129:40011)
I20260620 01:59:52.230454 26296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:52.341085 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 26174
I20260620 01:59:52.358572 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 25911
I20260620 01:59:52.367920 25974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:52.499854 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 25911
I20260620 01:59:52.522167 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 25224
I20260620 01:59:52.523774 25288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:52.657897 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 25224
W20260620 01:59:52.680291 25316 connection.cc:573] server connection from 127.0.0.1:41830 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 01:59:52.680708 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 25297
I20260620 01:59:52.682662 25360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:52.812748 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 25297
I20260620 01:59:52.833772 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 25567
I20260620 01:59:52.835376 25736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 01:59:52.972779 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 25567
Shutting down hive metastore.
2026-06-20T01:59:53Z chronyd exiting
2026-06-20T01:59:53Z chronyd exiting
2026-06-20T01:59:53Z chronyd exiting
2026-06-20T01:59:53Z chronyd exiting
2026-06-20T01:59:53Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/5 (35908 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/6
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 01:59:53 dist-test-slave-wg14 krb5kdc[26328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 01:59:53 dist-test-slave-wg14 krb5kdc[26328](info): set up 2 sockets
Jun 20 01:59:53 dist-test-slave-wg14 krb5kdc[26328](info): commencing operation
krb5kdc: starting...
W20260620 01:59:55.628405 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.151s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 01:59:55 dist-test-slave-wg14 krb5kdc[26328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:59:56.004608 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:39125
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:59:56.472342 26342 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:56.472878 26342 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:56.473055 26342 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:56.485096 26342 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:59:56.485267 26342 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:56.485354 26342 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:59:56.485425 26342 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:59:56.499071 26342 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:39125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:56.501293 26342 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:56.503180 26342 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:56.514189 26348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:56.514148 26347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:56.515501 26350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:56.515981 26342 server_base.cc:1061] running on GCE node
I20260620 01:59:56.516983 26342 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:59:56.517170 26342 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:59:56.517256 26342 hybrid_clock.cc:648] HybridClock initialized: now 1781920796517234 us; error 0 us; skew 500 ppm
I20260620 01:59:56.517593 26342 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:59:56 dist-test-slave-wg14 krb5kdc[26328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920796, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:59:56.521715 26342 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 01:59:56.524616 26342 webserver.cc:533] Webserver started at http://127.20.1.190:35633/ using document root <none> and password file <none>
I20260620 01:59:56.525576 26342 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:56.525899 26342 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:56.526952 26342 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:59:56.531106 26342 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "0a201db9764d44e491d7e825d5fbb897"
format_stamp: "Formatted at 2026-06-20 01:59:56 on dist-test-slave-wg14"
I20260620 01:59:56.532464 26342 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "0a201db9764d44e491d7e825d5fbb897"
format_stamp: "Formatted at 2026-06-20 01:59:56 on dist-test-slave-wg14"
I20260620 01:59:56.541875 26342 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260620 01:59:56.547935 26356 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:56.550552 26342 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260620 01:59:56.551072 26342 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "0a201db9764d44e491d7e825d5fbb897"
format_stamp: "Formatted at 2026-06-20 01:59:56 on dist-test-slave-wg14"
I20260620 01:59:56.551548 26342 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:56.570772 26342 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:56.578199 26342 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:56.578606 26342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:56.619028 26342 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:39125
I20260620 01:59:56.619088 26407 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:39125 every 8 connection(s)
I20260620 01:59:56.621543 26342 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 01:59:56.625177 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 26342
I20260620 01:59:56.625463 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 01:59:56.628681 26408 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:59:56.636467 26408 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:59:56 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920796, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.189@KRBTEST.COM" created.
I20260620 01:59:56.679582 26408 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:59:56.681552 26359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36161: connect: Connection refused (error 111)
W20260620 01:59:56.685868 26408 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36161: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36161: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:59:56.735373 26408 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } attempt: 1
W20260620 01:59:56.738260 26408 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36161: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36161: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:59:56.784389 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:36161
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:59:56.806651 26408 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } attempt: 2
W20260620 01:59:56.809828 26408 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36161: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36161: connect: Connection refused (error 111)
I20260620 01:59:56.901283 26408 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } attempt: 3
W20260620 01:59:56.904237 26408 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36161: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36161: connect: Connection refused (error 111)
I20260620 01:59:57.047554 26408 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } attempt: 4
W20260620 01:59:57.051036 26408 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36161: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36161: connect: Connection refused (error 111)
W20260620 01:59:57.210968 26417 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:57.211572 26417 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:57.211704 26417 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:57.222879 26417 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:59:57.223039 26417 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:57.223100 26417 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:59:57.223196 26417 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:59:57.236645 26417 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:36161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:57.238814 26417 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:57.240900 26417 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:57.253185 26422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:57.253194 26423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:57.254534 26425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:57.254937 26417 server_base.cc:1061] running on GCE node
I20260620 01:59:57.255963 26417 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:59:57.256186 26417 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:59:57.256294 26417 hybrid_clock.cc:648] HybridClock initialized: now 1781920797256266 us; error 0 us; skew 500 ppm
I20260620 01:59:57.256796 26417 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:59:57 dist-test-slave-wg14 krb5kdc[26328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920797, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:59:57.262439 26417 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 01:59:57.265836 26417 webserver.cc:533] Webserver started at http://127.20.1.189:36579/ using document root <none> and password file <none>
I20260620 01:59:57.267067 26417 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:57.267513 26417 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:57.268055 26417 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:59:57.271212 26417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "13e2e76c707d40b38be13fd660b098b1"
format_stamp: "Formatted at 2026-06-20 01:59:57 on dist-test-slave-wg14"
I20260620 01:59:57.272455 26417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "13e2e76c707d40b38be13fd660b098b1"
format_stamp: "Formatted at 2026-06-20 01:59:57 on dist-test-slave-wg14"
I20260620 01:59:57.280243 26417 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 01:59:57.285903 26431 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:57.288628 26417 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 01:59:57.288938 26417 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "13e2e76c707d40b38be13fd660b098b1"
format_stamp: "Formatted at 2026-06-20 01:59:57 on dist-test-slave-wg14"
I20260620 01:59:57.289227 26417 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:57.316866 26417 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:57.322124 26417 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:57.322477 26417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:57.349967 26417 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:36161
I20260620 01:59:57.349970 26482 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:36161 every 8 connection(s)
I20260620 01:59:57.350414 26408 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } attempt: 5
I20260620 01:59:57.352465 26417 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 01:59:57.362308 26483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:59:57.362715 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 26417
I20260620 01:59:57.363005 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 01:59:57.373087 26483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:59:57 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920796, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:59:57 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920797, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 01:59:57.394167 26408 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:59:57.397239 26408 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44219: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44219: connect: Connection refused (error 111)
I20260620 01:59:57.400027 26483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.188@KRBTEST.COM" created.
Jun 20 01:59:57 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920797, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:59:57.416081 26483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } has no permanent_uuid. Determining permanent_uuid...
W20260620 01:59:57.417724 26432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44219: connect: Connection refused (error 111)
W20260620 01:59:57.420013 26483 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44219: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44219: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:59:57.448580 26408 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } attempt: 1
W20260620 01:59:57.452051 26408 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44219: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44219: connect: Connection refused (error 111)
I20260620 01:59:57.469597 26483 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } attempt: 1
W20260620 01:59:57.473240 26483 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44219: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44219: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:59:57.519599 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:44219
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 01:59:57.520402 26408 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } attempt: 2
W20260620 01:59:57.523378 26408 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44219: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44219: connect: Connection refused (error 111)
I20260620 01:59:57.541606 26483 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } attempt: 2
W20260620 01:59:57.545959 26483 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44219: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44219: connect: Connection refused (error 111)
I20260620 01:59:57.629714 26408 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } attempt: 3
W20260620 01:59:57.632874 26408 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44219: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44219: connect: Connection refused (error 111)
I20260620 01:59:57.637526 26483 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } attempt: 3
W20260620 01:59:57.641772 26483 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44219: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44219: connect: Connection refused (error 111)
I20260620 01:59:57.785149 26483 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } attempt: 4
W20260620 01:59:57.787998 26483 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44219: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44219: connect: Connection refused (error 111)
I20260620 01:59:57.810222 26408 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } attempt: 4
W20260620 01:59:57.812621 26408 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44219: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44219: connect: Connection refused (error 111)
W20260620 01:59:57.983841 26493 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:57.984292 26493 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:57.984490 26493 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:57.994993 26493 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 01:59:57.995165 26493 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:57.995262 26493 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 01:59:57.995326 26493 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 01:59:58.009194 26493 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:44219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:58.011449 26493 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:58.013302 26493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:58.027405 26498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:58.028128 26499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:58.028642 26501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:58.029596 26493 server_base.cc:1061] running on GCE node
I20260620 01:59:58.030683 26493 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:59:58.030889 26493 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:59:58.031009 26493 hybrid_clock.cc:648] HybridClock initialized: now 1781920798030976 us; error 0 us; skew 500 ppm
I20260620 01:59:58.031499 26493 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:59:58 dist-test-slave-wg14 krb5kdc[26328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920798, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:59:58.037014 26493 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 01:59:58.040295 26493 webserver.cc:533] Webserver started at http://127.20.1.188:35173/ using document root <none> and password file <none>
I20260620 01:59:58.041718 26493 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:58.041997 26493 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:58.042574 26493 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:59:58.047442 26493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "0034969e2e3849048c79eea477475df3"
format_stamp: "Formatted at 2026-06-20 01:59:58 on dist-test-slave-wg14"
I20260620 01:59:58.048678 26493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "0034969e2e3849048c79eea477475df3"
format_stamp: "Formatted at 2026-06-20 01:59:58 on dist-test-slave-wg14"
I20260620 01:59:58.058569 26493 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20260620 01:59:58.064013 26507 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:58.066306 26493 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260620 01:59:58.066696 26493 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "0034969e2e3849048c79eea477475df3"
format_stamp: "Formatted at 2026-06-20 01:59:58 on dist-test-slave-wg14"
I20260620 01:59:58.066968 26493 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:58.087389 26483 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } attempt: 5
I20260620 01:59:58.089982 26408 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } attempt: 5
W20260620 01:59:58.091372 26483 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44219: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44219: connect: Connection refused (error 111)
W20260620 01:59:58.093189 26408 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:44219: Network error: Client connection negotiation failed: client connection to 127.20.1.188:44219: connect: Connection refused (error 111)
I20260620 01:59:58.096897 26493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:58.106016 26493 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:58.106348 26493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:58.142864 26558 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:44219 every 8 connection(s)
I20260620 01:59:58.142864 26493 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:44219
I20260620 01:59:58.145946 26493 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 01:59:58.149343 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 26493
I20260620 01:59:58.149662 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 01:59:58.154089 26559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 01:59:58.164844 26559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:59:58 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:59:58 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920798, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:59:58 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:59:58.210835 26559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:59:58 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920798, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 01:59:58 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:59:58.239506 26559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 01:59:58 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920798, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:59:58.270087 26559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3: Bootstrap starting.
I20260620 01:59:58.275347 26559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3: Neither blocks nor log segments found. Creating new log.
I20260620 01:59:58.277853 26559 log.cc:826] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3: Log is configured to *not* fsync() on all Append() calls
I20260620 01:59:58.282975 26559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3: No bootstrap required, opened a new log
I20260620 01:59:58.292119 26559 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } }
I20260620 01:59:58.293284 26559 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:59:58.294029 26559 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0034969e2e3849048c79eea477475df3, State: Initialized, Role: FOLLOWER
I20260620 01:59:58.295118 26559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } }
I20260620 01:59:58.296693 26568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } }
I20260620 01:59:58.297232 26568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:58.298332 26559 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 01:59:58.317026 26579 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:59:58.317384 26579 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:59:58.309839 26493 master_runner.cc:187] Error getting master registration for 127.20.1.190:39125: OK, instance_id { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" instance_seqno: 1781920796608495 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:59:58.320088 26493 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:59:58.360800 26513 tablet.cc:2406] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:59:58.375636 26568 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 01:59:58.376194 26568 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } }
I20260620 01:59:58.378582 26568 leader_election.cc:290] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a201db9764d44e491d7e825d5fbb897 (127.20.1.190:39125), 13e2e76c707d40b38be13fd660b098b1 (127.20.1.189:36161)
I20260620 01:59:58.617789 26408 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } attempt: 6
Jun 20 01:59:58 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920796, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:59:58.638708 26483 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } attempt: 6
I20260620 01:59:58.653695 26408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897: Bootstrap starting.
I20260620 01:59:58.661000 26408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897: Neither blocks nor log segments found. Creating new log.
Jun 20 01:59:58 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920797, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 01:59:58.671396 26408 log.cc:826] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897: Log is configured to *not* fsync() on all Append() calls
I20260620 01:59:58.677448 26408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897: No bootstrap required, opened a new log
I20260620 01:59:58.687883 26408 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } }
I20260620 01:59:58.688309 26408 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:59:58.688476 26408 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a201db9764d44e491d7e825d5fbb897, State: Initialized, Role: FOLLOWER
I20260620 01:59:58.689426 26408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } }
I20260620 01:59:58.690632 26585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } }
I20260620 01:59:58.690820 26483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1: Bootstrap starting.
I20260620 01:59:58.691099 26585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:58.692691 26408 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:59:58.692667 26383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0034969e2e3849048c79eea477475df3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a201db9764d44e491d7e825d5fbb897" is_pre_election: true
I20260620 01:59:58.693461 26383 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0034969e2e3849048c79eea477475df3 in term 0.
I20260620 01:59:58.694780 26511 leader_election.cc:304] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0034969e2e3849048c79eea477475df3, 0a201db9764d44e491d7e825d5fbb897; no voters:
I20260620 01:59:58.695845 26568 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 01:59:58.696134 26568 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 01:59:58.696277 26568 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:59:58.698565 26483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1: Neither blocks nor log segments found. Creating new log.
I20260620 01:59:58.699517 26568 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } }
I20260620 01:59:58.700345 26483 log.cc:826] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1: Log is configured to *not* fsync() on all Append() calls
I20260620 01:59:58.700735 26568 leader_election.cc:290] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [CANDIDATE]: Term 1 election: Requested vote from peers 0a201db9764d44e491d7e825d5fbb897 (127.20.1.190:39125), 13e2e76c707d40b38be13fd660b098b1 (127.20.1.189:36161)
I20260620 01:59:58.701186 26383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0034969e2e3849048c79eea477475df3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a201db9764d44e491d7e825d5fbb897"
I20260620 01:59:58.701663 26383 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:59:58.706666 26383 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0034969e2e3849048c79eea477475df3 in term 1.
I20260620 01:59:58.707937 26511 leader_election.cc:304] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0034969e2e3849048c79eea477475df3, 0a201db9764d44e491d7e825d5fbb897; no voters:
I20260620 01:59:58.708455 26568 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 01:59:58.709146 26568 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [term 1 LEADER]: Becoming Leader. State: Replica: 0034969e2e3849048c79eea477475df3, State: Running, Role: LEADER
I20260620 01:59:58.709939 26568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } }
I20260620 01:59:58.712978 26483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1: No bootstrap required, opened a new log
I20260620 01:59:58.724630 26483 raft_consensus.cc:359] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } }
I20260620 01:59:58.725307 26483 raft_consensus.cc:385] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 01:59:58.725484 26483 raft_consensus.cc:740] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e2e76c707d40b38be13fd660b098b1, State: Initialized, Role: FOLLOWER
I20260620 01:59:58.726687 26483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } }
I20260620 01:59:58.727665 26590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0034969e2e3849048c79eea477475df3. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } }
I20260620 01:59:58.728085 26590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:58.728883 26597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } }
I20260620 01:59:58.729822 26597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:58.730418 26483 sys_catalog.cc:565] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 01:59:58.730510 26457 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0034969e2e3849048c79eea477475df3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e2e76c707d40b38be13fd660b098b1"
I20260620 01:59:58.731223 26457 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 01:59:58.730428 26458 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0034969e2e3849048c79eea477475df3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e2e76c707d40b38be13fd660b098b1" is_pre_election: true
I20260620 01:59:58.738857 26457 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0034969e2e3849048c79eea477475df3 in term 1.
I20260620 01:59:58.740666 26598 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 01:59:58.743719 26598 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 01:59:58.767207 26457 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [term 1 FOLLOWER]: Refusing update from remote peer 0034969e2e3849048c79eea477475df3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:59:58.767320 26383 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [term 1 FOLLOWER]: Refusing update from remote peer 0034969e2e3849048c79eea477475df3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 01:59:58.768903 26568 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:59:58.769824 26568 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 01:59:58.794593 26600 mvcc.cc:204] Tried to move back new op lower bound from 7298747591731232768 to 7298747591524159488. Current Snapshot: MvccSnapshot[applied={T|T < 7298747591731232768}]
I20260620 01:59:58.794417 26585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0034969e2e3849048c79eea477475df3. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } }
I20260620 01:59:58.799669 26585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:58.803831 26568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } }
I20260620 01:59:58.804518 26568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:58.807864 26597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0034969e2e3849048c79eea477475df3. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } }
I20260620 01:59:58.808260 26597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:58.816280 26598 catalog_manager.cc:1382] Generated new cluster ID: 07367f4431254cd49e634ff1773da97b
I20260620 01:59:58.816504 26598 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
W20260620 01:59:58.827292 26437 tablet.cc:2406] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 01:59:58.832022 26590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } }
I20260620 01:59:58.833077 26590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: This master's current role is: LEADER
I20260620 01:59:58.833254 26585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } }
I20260620 01:59:58.833576 26585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:58.837114 26597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } }
I20260620 01:59:58.837481 26597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 01:59:58.841385 26342 master_runner.cc:187] Error getting master registration for 127.20.1.189:36161: OK, instance_id { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" instance_seqno: 1781920797342754 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 01:59:58.849640 26342 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 01:59:58.855093 26605 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 01:59:58.857596 26605 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 01:59:58.876514 26598 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 01:59:58.879320 26598 catalog_manager.cc:1539] Loading token signing keys...
I20260620 01:59:58.906497 26598 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3: Generated new TSK 0
I20260620 01:59:58.908114 26598 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 01:59:58.910935 26623 catalog_manager.cc:1294] Loaded cluster ID: 07367f4431254cd49e634ff1773da97b
I20260620 01:59:58.911165 26623 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1: loading cluster ID for follower catalog manager: success
I20260620 01:59:58.916078 26623 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1: acquiring CA information for follower catalog manager: success
I20260620 01:59:58.920621 26623 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 01:59:59.022774 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.101s user 0.000s sys 0.030s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 01:59:59.097187 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 01:59:59.581331 26630 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 01:59:59.581810 26630 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 01:59:59.581983 26630 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 01:59:59.593200 26630 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 01:59:59.593466 26630 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 01:59:59.607460 26630 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 01:59:59.609596 26630 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 01:59:59.611469 26630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 01:59:59.642025 26635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:59.644337 26636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 01:59:59.647624 26638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 01:59:59.648571 26630 server_base.cc:1061] running on GCE node
I20260620 01:59:59.649570 26630 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 01:59:59.649842 26630 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 01:59:59.649950 26630 hybrid_clock.cc:648] HybridClock initialized: now 1781920799649918 us; error 0 us; skew 500 ppm
I20260620 01:59:59.650367 26630 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 01:59:59 dist-test-slave-wg14 krb5kdc[26328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920799, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 01:59:59.656575 26630 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 01:59:59.659866 26630 webserver.cc:533] Webserver started at http://127.20.1.129:44191/ using document root <none> and password file <none>
I20260620 01:59:59.660940 26630 fs_manager.cc:362] Metadata directory not provided
I20260620 01:59:59.661128 26630 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 01:59:59.661597 26630 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 01:59:59.665951 26630 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "fa127c0a78774c728da9905a1a6a007e"
format_stamp: "Formatted at 2026-06-20 01:59:59 on dist-test-slave-wg14"
I20260620 01:59:59.667379 26630 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "fa127c0a78774c728da9905a1a6a007e"
format_stamp: "Formatted at 2026-06-20 01:59:59 on dist-test-slave-wg14"
I20260620 01:59:59.677443 26630 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260620 01:59:59.682981 26644 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:59.685258 26630 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260620 01:59:59.685503 26630 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "fa127c0a78774c728da9905a1a6a007e"
format_stamp: "Formatted at 2026-06-20 01:59:59 on dist-test-slave-wg14"
I20260620 01:59:59.685738 26630 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 01:59:59.709056 26630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 01:59:59.714711 26630 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 01:59:59.715037 26630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 01:59:59.716358 26630 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 01:59:59.719033 26630 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 01:59:59.719233 26630 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 01:59:59.719389 26630 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 01:59:59.719467 26630 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 01:59:59.777926 26756 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:43381 every 8 connection(s)
I20260620 01:59:59.777916 26630 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:43381
I20260620 01:59:59.779897 26630 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 01:59:59.780905 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 26630
I20260620 01:59:59.781158 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 01:59:59 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920799, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 01:59:59 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920799, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 01:59:59 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920799, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 01:59:59.835222 26757 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44219
I20260620 01:59:59.835256 26758 heartbeater.cc:344] Connected to a master server at 127.20.1.190:39125
I20260620 01:59:59.835816 26757 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:59.835819 26758 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:59.836975 26759 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36161
I20260620 01:59:59.837226 26759 heartbeater.cc:461] Registering TS with master...
I20260620 01:59:59.837512 26758 heartbeater.cc:507] Master 127.20.1.190:39125 requested a full tablet report, sending...
I20260620 01:59:59.837540 26757 heartbeater.cc:507] Master 127.20.1.188:44219 requested a full tablet report, sending...
I20260620 01:59:59.837898 26759 heartbeater.cc:507] Master 127.20.1.189:36161 requested a full tablet report, sending...
I20260620 01:59:59.840677 26524 ts_manager.cc:194] Registered new tserver with Master: fa127c0a78774c728da9905a1a6a007e (127.20.1.129:43381)
I20260620 01:59:59.840665 26372 ts_manager.cc:194] Registered new tserver with Master: fa127c0a78774c728da9905a1a6a007e (127.20.1.129:43381)
I20260620 01:59:59.840720 26447 ts_manager.cc:194] Registered new tserver with Master: fa127c0a78774c728da9905a1a6a007e (127.20.1.129:43381)
I20260620 01:59:59.843035 26524 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:53661
I20260620 01:59:59.859973 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 01:59:59.861406 26605 catalog_manager.cc:1294] Loaded cluster ID: 07367f4431254cd49e634ff1773da97b
I20260620 01:59:59.861582 26605 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897: loading cluster ID for follower catalog manager: success
I20260620 01:59:59.865918 26605 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897: acquiring CA information for follower catalog manager: success
I20260620 01:59:59.868912 26605 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for kudu/127.20.1.187@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.187@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.187 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.187 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.187@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.187@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.187 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.187 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:00:00.010252 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:44323
--webserver_interface=127.20.1.187
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.187
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219,127.20.1.187:44323 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:00:00.552204 26771 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:00.552635 26771 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:00.552795 26771 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:00.565708 26771 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:00:00.565896 26771 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:00.565980 26771 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:00:00.566049 26771 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:00:00.585966 26771 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219,127.20.1.187:44323
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.187
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:44323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.20.1.187
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:00.588498 26771 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:00.590734 26771 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:00.607746 26776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:00.608570 26777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:00.614235 26779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:00.614804 26771 server_base.cc:1061] running on GCE node
I20260620 02:00:00.616089 26771 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:00:00.616317 26771 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:00:00.616452 26771 hybrid_clock.cc:648] HybridClock initialized: now 1781920800616423 us; error 0 us; skew 500 ppm
I20260620 02:00:00.616896 26771 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:00 dist-test-slave-wg14 krb5kdc[26328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920800, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:00.624855 26771 init.cc:377] Logged in from keytab as kudu/127.20.1.187@KRBTEST.COM (short username kudu)
I20260620 02:00:00.629904 26771 webserver.cc:533] Webserver started at http://127.20.1.187:32981/ using document root <none> and password file <none>
I20260620 02:00:00.631107 26771 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:00.631305 26771 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:00.631771 26771 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:00:00.636982 26771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/data/instance:
uuid: "e32730b591aa4c5b8f2f2aa7ce249503"
format_stamp: "Formatted at 2026-06-20 02:00:00 on dist-test-slave-wg14"
I20260620 02:00:00.638286 26771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/wal/instance:
uuid: "e32730b591aa4c5b8f2f2aa7ce249503"
format_stamp: "Formatted at 2026-06-20 02:00:00 on dist-test-slave-wg14"
I20260620 02:00:00.648545 26771 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.010s sys 0.000s
I20260620 02:00:00.655821 26785 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:00.658567 26771 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.001s sys 0.002s
I20260620 02:00:00.658813 26771 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "e32730b591aa4c5b8f2f2aa7ce249503"
format_stamp: "Formatted at 2026-06-20 02:00:00 on dist-test-slave-wg14"
I20260620 02:00:00.659052 26771 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:00.699347 26771 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:00.708942 26771 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:00.709352 26771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:00.744129 26771 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:44323
I20260620 02:00:00.744129 26836 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:44323 every 8 connection(s)
I20260620 02:00:00.746229 26771 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 02:00:00.751461 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 26771
I20260620 02:00:00.755595 26837 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:00:00.765765 26837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:00 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
Jun 20 02:00:00 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920800, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:00:00.832823 26837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:00 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920800, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:00:00.847119 26757 heartbeater.cc:499] Master 127.20.1.188:44219 was elected leader, sending a full tablet report...
I20260620 02:00:00.856014 26837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:00 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920800, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:00:00.875766 26837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:00 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920800, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
I20260620 02:00:00.905054 26837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: Bootstrap starting.
I20260620 02:00:00.908958 26837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: Neither blocks nor log segments found. Creating new log.
I20260620 02:00:00.910362 26837 log.cc:826] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: Log is configured to *not* fsync() on all Append() calls
I20260620 02:00:00.919795 26837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: No bootstrap required, opened a new log
I20260620 02:00:00.927296 26837 raft_consensus.cc:359] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } }
I20260620 02:00:00.928026 26837 raft_consensus.cc:385] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:00:00.928174 26837 raft_consensus.cc:740] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32730b591aa4c5b8f2f2aa7ce249503, State: Initialized, Role: FOLLOWER
I20260620 02:00:00.929111 26837 consensus_queue.cc:260] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } }
I20260620 02:00:00.931211 26846 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } } }
I20260620 02:00:00.932914 26837 sys_catalog.cc:565] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:00:00.932971 26846 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 02:00:00.949422 26857 catalog_manager.cc:1593] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:00:00.949638 26857 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:00:00.962324 26771 master_runner.cc:429] Detected that this master e32730b591aa4c5b8f2f2aa7ce249503 is joining an existing cluster
I20260620 02:00:00.962978 26771 catalog_manager.cc:1024] Shutting down the Catalog Manager
W20260620 02:00:00.963418 26791 tablet.cc:2406] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:00:00.964797 26771 raft_consensus.cc:2243] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 02:00:00.965598 26771 raft_consensus.cc:2272] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 02:00:00.965812 26771 tablet_replica.cc:333] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: stopping tablet replica
I20260620 02:00:00.975909 26771 master_runner.cc:434] Initiating AddMaster RPC to add 127.20.1.187:44323
Jun 20 02:00:01 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920798, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
I20260620 02:00:01.112687 26524 catalog_manager.cc:7201] Initiating ChangeConfig request to add master 127.20.1.187:44323: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503"
member_type: NON_VOTER
last_known_addr {
host: "127.20.1.187"
port: 44323
}
attrs {
promote: true
}
}
dest_uuid: "0034969e2e3849048c79eea477475df3"
cas_config_opid_index: -1
I20260620 02:00:01.114200 26524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: true } }
I20260620 02:00:01.120465 26383 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [term 1 FOLLOWER]: Refusing update from remote peer 0034969e2e3849048c79eea477475df3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260620 02:00:01.120918 26509 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 -> Peer e32730b591aa4c5b8f2f2aa7ce249503 (127.20.1.187:44323): Couldn't send request to peer e32730b591aa4c5b8f2f2aa7ce249503. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260620 02:00:01.121842 26568 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 02:00:01.126137 26457 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [term 1 FOLLOWER]: Refusing update from remote peer 0034969e2e3849048c79eea477475df3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 02:00:01.128028 26568 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 02:00:01.130172 26870 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e32730b591aa4c5b8f2f2aa7ce249503 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: true } } }
I20260620 02:00:01.131083 26383 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e32730b591aa4c5b8f2f2aa7ce249503 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: true } } }
I20260620 02:00:01.135545 26870 catalog_manager.cc:7205] Successfully completed master ChangeConfig request to add master 127.20.1.187:44323
I20260620 02:00:01.138015 26771 master.cc:562] Master@127.20.1.187:44323 shutting down...
I20260620 02:00:01.151969 26568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: true } } }
I20260620 02:00:01.152384 26568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:01.155900 26873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: true } } }
I20260620 02:00:01.158756 26873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:01.160100 26870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: true } } }
I20260620 02:00:01.160372 26870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:01.167706 26568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: true } } }
I20260620 02:00:01.168260 26568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:01.168901 26457 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e32730b591aa4c5b8f2f2aa7ce249503 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: true } } }
I20260620 02:00:01.194228 26877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: true } } }
I20260620 02:00:01.196615 26877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 02:00:01.208276 26326 connection.cc:573] client connection to 127.20.1.187:44323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:00:01.217444 26771 master.cc:584] Master@127.20.1.187:44323 shutdown complete.
I20260620 02:00:01.217691 26771 master_runner.cc:306] Clearing existing system tablet
I20260620 02:00:01.218382 26771 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:01.218565 26771 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:01.222388 26771 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 02:00:01.226944 26879 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:01.230787 26771 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:01.231293 26771 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260620 02:00:01.231432 26771 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "e32730b591aa4c5b8f2f2aa7ce249503"
format_stamp: "Formatted at 2026-06-20 02:00:00 on dist-test-slave-wg14"
I20260620 02:00:01.238143 26771 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:00:01.248042 26771 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 02:00:01.248333 26771 log.cc:1199] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: Deleting WAL directory at /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260620 02:00:01.248888 26771 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: Deleting consensus metadata
I20260620 02:00:01.249964 26771 master_runner.cc:316] Copying system tablet from 127.20.1.188:44219
I20260620 02:00:01.255264 26771 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: tablet copy: Beginning tablet copy session from remote peer at address 127.20.1.188:44219
I20260620 02:00:01.282358 26544 tablet_copy_service.cc:140] P 0034969e2e3849048c79eea477475df3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e32730b591aa4c5b8f2f2aa7ce249503 ({username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:51294)
I20260620 02:00:01.282681 26544 tablet_copy_service.cc:161] P 0034969e2e3849048c79eea477475df3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e32730b591aa4c5b8f2f2aa7ce249503 at {username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:51294: session id = e32730b591aa4c5b8f2f2aa7ce249503-00000000000000000000000000000000
I20260620 02:00:01.295917 26544 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 02:00:01.299499 26771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 02:00:01.308954 26771 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: tablet copy: Starting download of 0 data blocks...
I20260620 02:00:01.309417 26771 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: tablet copy: Starting download of 1 WAL segments...
I20260620 02:00:01.315881 26771 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 02:00:01.325850 26544 tablet_copy_service.cc:342] P 0034969e2e3849048c79eea477475df3: Request end of tablet copy session e32730b591aa4c5b8f2f2aa7ce249503-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:51294
I20260620 02:00:01.326260 26544 tablet_copy_service.cc:434] P 0034969e2e3849048c79eea477475df3: ending tablet copy session e32730b591aa4c5b8f2f2aa7ce249503-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e32730b591aa4c5b8f2f2aa7ce249503
I20260620 02:00:01.356758 26771 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:01.366044 26886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:01.376284 26887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:01.377156 26889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:01.377266 26771 server_base.cc:1061] running on GCE node
I20260620 02:00:01.377827 26771 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:00:01.377934 26771 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:00:01.378017 26771 hybrid_clock.cc:648] HybridClock initialized: now 1781920801377971 us; error 0 us; skew 500 ppm
I20260620 02:00:01.378237 26771 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:01 dist-test-slave-wg14 krb5kdc[26328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920801, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:01.380724 26771 init.cc:377] Logged in from keytab as kudu/127.20.1.187@KRBTEST.COM (short username kudu)
I20260620 02:00:01.382385 26771 webserver.cc:533] Webserver started at http://127.20.1.187:45319/ using document root <none> and password file <none>
I20260620 02:00:01.382846 26771 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:01.382946 26771 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:01.389453 26771 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:00:01.395433 26895 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:01.396198 26771 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 02:00:01.396394 26771 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "e32730b591aa4c5b8f2f2aa7ce249503"
format_stamp: "Formatted at 2026-06-20 02:00:00 on dist-test-slave-wg14"
I20260620 02:00:01.396555 26771 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:01.419332 26771 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:01.424177 26771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:01.460301 26771 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:44323
I20260620 02:00:01.460338 26946 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:44323 every 8 connection(s)
I20260620 02:00:01.461498 26771 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 02:00:01.467674 26947 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:00:01.471045 26947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: Bootstrap starting.
I20260620 02:00:01.504654 26947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:00:01.505383 26947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: Bootstrap complete.
I20260620 02:00:01.506844 26947 raft_consensus.cc:359] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: true } }
I20260620 02:00:01.507220 26947 raft_consensus.cc:740] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e32730b591aa4c5b8f2f2aa7ce249503, State: Initialized, Role: LEARNER
I20260620 02:00:01.507587 26947 consensus_queue.cc:260] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: true } }
I20260620 02:00:01.513531 26947 sys_catalog.cc:565] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:00:01.514271 26951 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: true } } }
I20260620 02:00:01.515261 26951 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:00:01.532142 26962 catalog_manager.cc:1294] Loaded cluster ID: 07367f4431254cd49e634ff1773da97b
I20260620 02:00:01.532274 26962 catalog_manager.cc:1587] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: loading cluster ID for follower catalog manager: success
I20260620 02:00:01.536906 26962 catalog_manager.cc:1609] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: acquiring CA information for follower catalog manager: success
I20260620 02:00:01.541981 26962 catalog_manager.cc:1637] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:00:01.570577 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 26630
Jun 20 02:00:01 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:00:01.600451 26752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:00:01.697633 26922 raft_consensus.cc:1217] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260620 02:00:01.698664 26951 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0034969e2e3849048c79eea477475df3. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: true } } }
I20260620 02:00:01.699153 26951 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:00:01.738056 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 26630
I20260620 02:00:01.756397 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:43381
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219
--webserver_port=44191
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219,127.20.1.187:44323 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:00:02.119632 26874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: true } } }
I20260620 02:00:02.120726 26874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:02.122059 26967 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3: attempting to promote NON_VOTER e32730b591aa4c5b8f2f2aa7ce249503 to VOTER
I20260620 02:00:02.123090 26967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } }
I20260620 02:00:02.127233 26457 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [term 1 FOLLOWER]: Refusing update from remote peer 0034969e2e3849048c79eea477475df3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:00:02.127358 26922 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [term 1 LEARNER]: Refusing update from remote peer 0034969e2e3849048c79eea477475df3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:00:02.128135 26874 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:00:02.128526 26383 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [term 1 FOLLOWER]: Refusing update from remote peer 0034969e2e3849048c79eea477475df3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:00:02.128582 26967 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:00:02.129437 26970 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:00:02.133865 26874 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e32730b591aa4c5b8f2f2aa7ce249503 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } } }
I20260620 02:00:02.136262 26383 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e32730b591aa4c5b8f2f2aa7ce249503 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } } }
I20260620 02:00:02.141748 26457 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e32730b591aa4c5b8f2f2aa7ce249503 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } } }
I20260620 02:00:02.149672 26970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } } }
I20260620 02:00:02.150238 26970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:02.149659 26967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } } }
I20260620 02:00:02.154347 26967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:02.154373 26970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } } }
I20260620 02:00:02.155824 26970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:02.158280 26972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } } }
I20260620 02:00:02.158600 26972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:02.158463 26973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } } }
I20260620 02:00:02.158738 26973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13e2e76c707d40b38be13fd660b098b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:02.160730 26874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } } }
I20260620 02:00:02.161101 26874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:02.163862 26922 raft_consensus.cc:2955] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e32730b591aa4c5b8f2f2aa7ce249503 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } } }
I20260620 02:00:02.169613 26951 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } } }
I20260620 02:00:02.169930 26951 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32730b591aa4c5b8f2f2aa7ce249503 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 02:00:02.375337 26966 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:02.375752 26966 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:02.375914 26966 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:02.390372 26966 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:02.390595 26966 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:00:02.408174 26966 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:43381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=44191
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219,127.20.1.187:44323
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:02.410151 26966 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:02.411754 26966 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:02.426766 26979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:02.426819 26978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:02.429543 26981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:02.429740 26966 server_base.cc:1061] running on GCE node
I20260620 02:00:02.431035 26966 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:00:02.431573 26966 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:00:02.431684 26966 hybrid_clock.cc:648] HybridClock initialized: now 1781920802431663 us; error 0 us; skew 500 ppm
I20260620 02:00:02.432111 26966 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:02 dist-test-slave-wg14 krb5kdc[26328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920802, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:02.439857 26966 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:00:02.450183 26966 webserver.cc:533] Webserver started at http://127.20.1.129:44191/ using document root <none> and password file <none>
I20260620 02:00:02.451486 26966 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:02.451666 26966 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:02.464205 26966 fs_manager.cc:714] Time spent opening directory manager: real 0.010s user 0.007s sys 0.004s
I20260620 02:00:02.470230 26987 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:02.472325 26966 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 02:00:02.472545 26966 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "fa127c0a78774c728da9905a1a6a007e"
format_stamp: "Formatted at 2026-06-20 01:59:59 on dist-test-slave-wg14"
I20260620 02:00:02.473423 26966 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:02.506522 26966 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:02.511739 26966 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:02.512063 26966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:02.513494 26966 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:00:02.515997 26966 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:00:02.516155 26966 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:02.516283 26966 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:00:02.516355 26966 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:02.586668 26966 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:43381
I20260620 02:00:02.586710 27099 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:43381 every 8 connection(s)
I20260620 02:00:02.588737 26966 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:00:02.594817 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 26966
Jun 20 02:00:02 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920802, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 02:00:02 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920802, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
Jun 20 02:00:02 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920802, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 02:00:02 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920802, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:00:02.717711 27100 heartbeater.cc:344] Connected to a master server at 127.20.1.187:44323
I20260620 02:00:02.718230 27100 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:02.719671 27100 heartbeater.cc:507] Master 127.20.1.187:44323 requested a full tablet report, sending...
I20260620 02:00:02.723114 26911 ts_manager.cc:194] Registered new tserver with Master: fa127c0a78774c728da9905a1a6a007e (127.20.1.129:43381)
I20260620 02:00:02.723189 27102 heartbeater.cc:344] Connected to a master server at 127.20.1.190:39125
I20260620 02:00:02.723414 27102 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:02.724115 27102 heartbeater.cc:507] Master 127.20.1.190:39125 requested a full tablet report, sending...
I20260620 02:00:02.729530 27103 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36161
I20260620 02:00:02.729771 27103 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:02.731123 27103 heartbeater.cc:507] Master 127.20.1.189:36161 requested a full tablet report, sending...
I20260620 02:00:02.733119 26372 ts_manager.cc:194] Re-registered known tserver with Master: fa127c0a78774c728da9905a1a6a007e (127.20.1.129:43381)
I20260620 02:00:02.733510 27101 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44219
I20260620 02:00:02.733706 27101 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:02.734000 26447 ts_manager.cc:194] Re-registered known tserver with Master: fa127c0a78774c728da9905a1a6a007e (127.20.1.129:43381)
I20260620 02:00:02.734565 27101 heartbeater.cc:507] Master 127.20.1.188:44219 requested a full tablet report, sending...
I20260620 02:00:02.738672 26524 ts_manager.cc:194] Re-registered known tserver with Master: fa127c0a78774c728da9905a1a6a007e (127.20.1.129:43381)
I20260620 02:00:02.739454 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 02:00:02.739986 26524 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:60769
I20260620 02:00:02.761272 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 02:00:02.761476 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 26342
I20260620 02:00:02.763054 26403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:00:02.999351 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 26342
I20260620 02:00:03.028298 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:39125
--webserver_interface=127.20.1.190
--webserver_port=35633
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219
--master_addresses=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219,127.20.1.187:44323 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:00:03.213801 26511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.190:39125: connect: Connection refused (error 111)
W20260620 02:00:03.216784 26511 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 -> Peer 0a201db9764d44e491d7e825d5fbb897 (127.20.1.190:39125): Couldn't send request to peer 0a201db9764d44e491d7e825d5fbb897. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:39125: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 02:00:03.322074 26493 master_runner.cc:187] Error getting master registration for 127.20.1.190:39125: Network error: Client connection negotiation failed: client connection to 127.20.1.190:39125: connect: Connection refused (error 111),
I20260620 02:00:03.322384 26493 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:00:03.632258 26874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } } }
I20260620 02:00:03.632773 26874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: This master's current role is: LEADER
W20260620 02:00:03.739157 27102 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:39125 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:39125: connect: Connection refused (error 111)
I20260620 02:00:03.746896 27101 heartbeater.cc:499] Master 127.20.1.188:44219 was elected leader, sending a full tablet report...
W20260620 02:00:03.773960 27114 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:03.774395 27114 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:03.774753 27114 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:03.786567 27114 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:00:03.786705 27114 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:03.786765 27114 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:00:03.786806 27114 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:00:03.802212 27114 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219,127.20.1.187:44323
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:39125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=35633
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:03.805605 27114 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:03.807426 27114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:03.820539 27125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:03.821708 27128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:03.821748 27126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:03.828114 27114 server_base.cc:1061] running on GCE node
I20260620 02:00:03.835417 27114 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:00:03.835685 27114 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:00:03.835805 27114 hybrid_clock.cc:648] HybridClock initialized: now 1781920803835783 us; error 0 us; skew 500 ppm
I20260620 02:00:03.836216 27114 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:03 dist-test-slave-wg14 krb5kdc[26328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920803, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:03.841780 27114 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:00:03.855188 27114 webserver.cc:533] Webserver started at http://127.20.1.190:35633/ using document root <none> and password file <none>
I20260620 02:00:03.856840 27114 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:03.857275 27114 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:03.875541 27114 fs_manager.cc:714] Time spent opening directory manager: real 0.015s user 0.006s sys 0.000s
I20260620 02:00:03.880105 27134 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:03.882146 27114 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:00:03.882400 27114 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "0a201db9764d44e491d7e825d5fbb897"
format_stamp: "Formatted at 2026-06-20 01:59:56 on dist-test-slave-wg14"
I20260620 02:00:03.883481 27114 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:03.921273 27114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:03.937508 27114 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:03.937934 27114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:04.001026 27114 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:39125
I20260620 02:00:04.001055 27185 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:39125 every 8 connection(s)
I20260620 02:00:04.003093 27114 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:00:04.014135 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 27114
I20260620 02:00:04.014521 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 26966
I20260620 02:00:04.035557 27186 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:00:04.040310 27186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897: Bootstrap starting.
I20260620 02:00:04.072693 27095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:00:04.098398 27186 log.cc:826] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897: Log is configured to *not* fsync() on all Append() calls
I20260620 02:00:04.118402 27186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:00:04.120783 27186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897: Bootstrap complete.
I20260620 02:00:04.139534 27186 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } }
I20260620 02:00:04.140472 27186 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a201db9764d44e491d7e825d5fbb897, State: Initialized, Role: FOLLOWER
I20260620 02:00:04.141691 27186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } }
I20260620 02:00:04.142972 27192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } } }
I20260620 02:00:04.143632 27192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:04.154925 27186 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:00:04.229537 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 26966
I20260620 02:00:04.238386 27205 catalog_manager.cc:1294] Loaded cluster ID: 07367f4431254cd49e634ff1773da97b
I20260620 02:00:04.243162 27205 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897: loading cluster ID for follower catalog manager: success
I20260620 02:00:04.260891 27205 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897: acquiring CA information for follower catalog manager: success
I20260620 02:00:04.266041 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:43381
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219
--webserver_port=44191
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219,127.20.1.187:44323 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:00:04.297041 27205 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:00:04.304240 27192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0034969e2e3849048c79eea477475df3. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } } }
I20260620 02:00:04.315882 27192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a201db9764d44e491d7e825d5fbb897 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 02:00:04 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920803, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:00:04.644526 27118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0034969e2e3849048c79eea477475df3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a201db9764d44e491d7e825d5fbb897" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 39125 } } peers { permanent_uuid: "13e2e76c707d40b38be13fd660b098b1" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "0034969e2e3849048c79eea477475df3" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 44219 } } peers { permanent_uuid: "e32730b591aa4c5b8f2f2aa7ce249503" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 44323 } attrs { promote: false } } }
I20260620 02:00:04.645128 27118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0034969e2e3849048c79eea477475df3 [sys.catalog]: This master's current role is: LEADER
W20260620 02:00:04.873662 27209 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:04.874138 27209 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:04.874343 27209 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:04.886200 27209 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:04.886430 27209 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:00:04.900650 27209 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:43381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=44191
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219,127.20.1.187:44323
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:04.903285 27209 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:04.905304 27209 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:04.919831 27220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:04.920125 27221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:04.923440 27223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:04.925246 27209 server_base.cc:1061] running on GCE node
I20260620 02:00:04.926029 27209 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:00:04.926322 27209 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:00:04.926419 27209 hybrid_clock.cc:648] HybridClock initialized: now 1781920804926397 us; error 0 us; skew 500 ppm
I20260620 02:00:04.926847 27209 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:04 dist-test-slave-wg14 krb5kdc[26328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920804, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:04.932000 27209 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:00:04.936380 27209 webserver.cc:533] Webserver started at http://127.20.1.129:44191/ using document root <none> and password file <none>
I20260620 02:00:04.938061 27209 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:04.938547 27209 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:04.950126 27209 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:00:04.955442 27229 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:04.960593 27209 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.005s sys 0.000s
I20260620 02:00:04.960868 27209 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "fa127c0a78774c728da9905a1a6a007e"
format_stamp: "Formatted at 2026-06-20 01:59:59 on dist-test-slave-wg14"
I20260620 02:00:04.963899 27209 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:04.988808 27209 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:04.993711 27209 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:04.994063 27209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:04.996317 27209 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:00:05.001789 27209 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:00:05.001935 27209 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:05.002879 27209 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:00:05.003014 27209 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:05.091971 27209 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:43381
I20260620 02:00:05.092064 27341 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:43381 every 8 connection(s)
I20260620 02:00:05.093860 27209 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:00:05.096342 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 27209
Jun 20 02:00:05 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920804, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 02:00:05 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920804, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
Jun 20 02:00:05 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920804, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 02:00:05 dist-test-slave-wg14 krb5kdc[26328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920804, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:00:05.198450 27342 heartbeater.cc:344] Connected to a master server at 127.20.1.187:44323
I20260620 02:00:05.199064 27342 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:05.203064 27344 heartbeater.cc:344] Connected to a master server at 127.20.1.190:39125
I20260620 02:00:05.203738 27344 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:05.205943 27344 heartbeater.cc:507] Master 127.20.1.190:39125 requested a full tablet report, sending...
I20260620 02:00:05.210513 27345 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36161
I20260620 02:00:05.211068 27345 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:05.209667 27342 heartbeater.cc:507] Master 127.20.1.187:44323 requested a full tablet report, sending...
I20260620 02:00:05.212414 27345 heartbeater.cc:507] Master 127.20.1.189:36161 requested a full tablet report, sending...
I20260620 02:00:05.216013 26447 ts_manager.cc:194] Re-registered known tserver with Master: fa127c0a78774c728da9905a1a6a007e (127.20.1.129:43381)
I20260620 02:00:05.216827 26911 ts_manager.cc:194] Re-registered known tserver with Master: fa127c0a78774c728da9905a1a6a007e (127.20.1.129:43381)
I20260620 02:00:05.216964 27151 ts_manager.cc:194] Registered new tserver with Master: fa127c0a78774c728da9905a1a6a007e (127.20.1.129:43381)
I20260620 02:00:05.217345 27343 heartbeater.cc:344] Connected to a master server at 127.20.1.188:44219
I20260620 02:00:05.217692 27343 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:05.218968 27343 heartbeater.cc:507] Master 127.20.1.188:44219 requested a full tablet report, sending...
I20260620 02:00:05.220714 26524 ts_manager.cc:194] Re-registered known tserver with Master: fa127c0a78774c728da9905a1a6a007e (127.20.1.129:43381)
I20260620 02:00:05.223019 26524 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:49583
W20260620 02:00:05.237968 20486 mini_kdc.cc:336] Time spent kdestroy: real 0.141s user 0.002s sys 0.039s
I20260620 02:00:05.242247 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 27209
W20260620 02:00:05.245040 27358 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:05.239403 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.20.1.129:43381)
0620 02:00:05.240047 (+ 644us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:05.240167 (+ 120us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:05.240225 (+ 58us) server_negotiation.cc:408] Connection header received
0620 02:00:05.240312 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:00:05.240320 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:00:05.240558 (+ 238us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:00:05.240764 (+ 206us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:00:05.242550 (+ 1786us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38880: BlockingRecv error: recv got EOF from 127.0.0.1:38880 (error 108)
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":332,"threads_started":1}
W20260620 02:00:05.246215 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 27209: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:43381: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:00:05.246423 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 27209
W20260620 02:00:05.246462 27359 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:05.243521 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:38888 (local address 127.20.1.129:43381)
0620 02:00:05.244521 (+ 1000us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:05.244527 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:05.244559 (+ 32us) server_negotiation.cc:408] Connection header received
0620 02:00:05.244638 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:00:05.244643 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:00:05.244727 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:00:05.244873 (+ 146us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:00:05.246110 (+ 1237us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38888: BlockingRecv error: recv got EOF from 127.0.0.1:38888 (error 108)
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":639,"threads_started":1}
I20260620 02:00:05.269763 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:43381
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219
--webserver_port=44191
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219,127.20.1.187:44323
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:00:05.803877 27360 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:05.804247 27360 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:05.804419 27360 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 02:00:05.804497 27360 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:05.814332 27360 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:05.814627 27360 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:00:05.833316 27360 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:43381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=44191
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219,127.20.1.187:44323
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:05.835753 27360 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:05.838554 27360 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:05.855329 27366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:05.855964 27365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:05.857729 27368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:05.858609 27360 server_base.cc:1061] running on GCE node
I20260620 02:00:05.859771 27360 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:00:05.859961 27360 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:00:05.860252 27360 hybrid_clock.cc:648] HybridClock initialized: now 1781920805860019 us; error 0 us; skew 500 ppm
I20260620 02:00:05.860724 27360 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:05 dist-test-slave-wg14 krb5kdc[26328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920805, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:05.868257 27360 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:00:05.870766 27360 webserver.cc:533] Webserver started at http://127.20.1.129:44191/ using document root <none> and password file <none>
I20260620 02:00:05.872023 27360 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:05.872452 27360 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:05.880844 27360 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:00:05.887948 27374 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:05.890479 27360 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260620 02:00:05.890735 27360 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "fa127c0a78774c728da9905a1a6a007e"
format_stamp: "Formatted at 2026-06-20 01:59:59 on dist-test-slave-wg14"
I20260620 02:00:05.892519 27360 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:05.930146 27360 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:05.936658 27360 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:05.937019 27360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:05.939424 27360 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:00:05.941941 27360 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:00:05.942076 27360 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:05.942219 27360 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:00:05.942309 27360 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 02:00:05.943425 27360 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 02:00:05.943527 27360 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 02:00:05.943620 27360 fault_injection.cc:59] injecting fault for kudu (pid 27360): FLAGS_fault_before_start (process will exit)
I20260620 02:00:05.970857 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:43381
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219
--webserver_port=44191
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219,127.20.1.187:44323 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:00:06.682626 27382 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:06.683960 27382 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:06.684172 27382 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:06.696722 27382 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:06.697026 27382 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:00:06.712733 27382 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:43381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=44191
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:39125,127.20.1.189:36161,127.20.1.188:44219,127.20.1.187:44323
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:06.715255 27382 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:06.717438 27382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:06.736651 27388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:06.736671 27387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:06.736562 27390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:06.737658 27382 server_base.cc:1061] running on GCE node
I20260620 02:00:06.738772 27382 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:00:06.738998 27382 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:00:06.739100 27382 hybrid_clock.cc:648] HybridClock initialized: now 1781920806739077 us; error 0 us; skew 500 ppm
I20260620 02:00:06.739521 27382 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:06 dist-test-slave-wg14 krb5kdc[26328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920806, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:06.744978 27382 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:00:06.750507 27382 webserver.cc:533] Webserver started at http://127.20.1.129:44191/ using document root <none> and password file <none>
I20260620 02:00:06.752035 27382 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:06.752192 27382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:06.763104 27382 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.005s sys 0.001s
I20260620 02:00:06.770081 27396 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:06.773279 27382 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:00:06.773523 27382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "fa127c0a78774c728da9905a1a6a007e"
format_stamp: "Formatted at 2026-06-20 01:59:59 on dist-test-slave-wg14"
I20260620 02:00:06.774578 27382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:06.815178 27382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:06.823740 27382 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:06.824749 27382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:06.827026 27382 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:00:06.829706 27382 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:00:06.829898 27382 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:06.830053 27382 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:00:06.830111 27382 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:06.891793 27382 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:43381
I20260620 02:00:06.891876 27508 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:43381 every 8 connection(s)
I20260620 02:00:06.893662 27382 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:00:06.901124 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 27382
I20260620 02:00:06.901563 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 27382
W20260620 02:00:06.933991 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 27382: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:43381: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:00:06.934171 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 27382
W20260620 02:00:06.959211 27515 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:06.898679 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:60839 (local address 127.20.1.189:36161)
0620 02:00:06.931932 (+ 33253us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:06.931939 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:06.931966 (+ 27us) server_negotiation.cc:408] Connection header received
0620 02:00:06.932057 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:00:06.932112 (+ 55us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:00:06.932203 (+ 91us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:00:06.932340 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:00:06.957946 (+ 25606us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:60839: BlockingRecv error: recv got EOF from 127.20.1.129:60839 (error 108)
Metrics: {"server-negotiator.queue_time_us":33054,"thread_start_us":33853,"threads_started":1}
W20260620 02:00:06.959211 27516 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:06.898675 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:42469 (local address 127.20.1.188:44219)
0620 02:00:06.918740 (+ 20065us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:06.918748 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:06.931879 (+ 13131us) server_negotiation.cc:408] Connection header received
0620 02:00:06.932026 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:00:06.932032 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:00:06.932125 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:00:06.932237 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:00:06.957832 (+ 25595us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:42469: BlockingRecv error: recv got EOF from 127.20.1.129:42469 (error 108)
Metrics: {"server-negotiator.queue_time_us":19869,"thread_start_us":20115,"threads_started":1}
W20260620 02:00:06.959246 27514 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:06.898684 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:40017 (local address 127.20.1.187:44323)
0620 02:00:06.937673 (+ 38989us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:06.937681 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:06.937715 (+ 34us) server_negotiation.cc:408] Connection header received
0620 02:00:06.937801 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:00:06.937808 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:00:06.938061 (+ 253us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:00:06.938409 (+ 348us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:00:06.958418 (+ 20009us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:40017: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":38443,"thread_start_us":39530,"threads_started":1}
I20260620 02:00:06.959517 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 27114
W20260620 02:00:06.961033 27523 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:06.929683 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:45731 (local address 127.20.1.190:39125)
0620 02:00:06.931752 (+ 2069us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:06.931759 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:06.935521 (+ 3762us) server_negotiation.cc:408] Connection header received
0620 02:00:06.935671 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:00:06.935687 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:00:06.935810 (+ 123us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:00:06.935974 (+ 164us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:00:06.959918 (+ 23944us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:45731: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":948,"thread_start_us":765,"threads_started":1}
W20260620 02:00:06.962926 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 27114: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:39125: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:00:06.963200 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 27114
I20260620 02:00:06.987221 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 26417
I20260620 02:00:06.991285 26478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:00:07.175985 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 26417
I20260620 02:00:07.201859 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 26493
I20260620 02:00:07.204111 26554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:00:07.374280 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 26493
I20260620 02:00:07.401223 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 26771
I20260620 02:00:07.402812 26942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:00:07.599074 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 26771
[ OK ] ExternalMiniClusterTest.TestAddMaster/6 (14164 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/7
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:00:07 dist-test-slave-wg14 krb5kdc[27531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:00:07 dist-test-slave-wg14 krb5kdc[27531](info): set up 2 sockets
Jun 20 02:00:07 dist-test-slave-wg14 krb5kdc[27531](info): commencing operation
krb5kdc: starting...
W20260620 02:00:10.420212 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.778s user 0.002s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 02:00:10 dist-test-slave-wg14 krb5kdc[27531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:00:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:00:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:00:10.881326 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.190: real 0.101s user 0.005s sys 0.019s
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:00:10.975416 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:36605
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:00:11.641127 27547 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:11.641602 27547 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:11.641763 27547 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:11.658892 27547 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:00:11.659076 27547 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:11.659196 27547 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:00:11.659277 27547 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:00:11.681130 27547 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:36605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:11.683246 27547 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:11.685011 27547 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:11.703365 27552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:11.703333 27553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:11.704885 27555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:11.705350 27547 server_base.cc:1061] running on GCE node
I20260620 02:00:11.706466 27547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:11.709533 27547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:11.713397 27547 hybrid_clock.cc:648] HybridClock initialized: now 1781920811713171 us; error 181 us; skew 500 ppm
I20260620 02:00:11.715214 27547 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:11 dist-test-slave-wg14 krb5kdc[27531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920811, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:11.720575 27547 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:00:11.723436 27547 webserver.cc:533] Webserver started at http://127.20.1.190:34087/ using document root <none> and password file <none>
I20260620 02:00:11.724390 27547 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:11.724527 27547 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:11.724946 27547 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:00:11.731012 27547 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "2e4c0714eed64d479e0e70654e1e6a10"
format_stamp: "Formatted at 2026-06-20 02:00:11 on dist-test-slave-wg14"
I20260620 02:00:11.732312 27547 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "2e4c0714eed64d479e0e70654e1e6a10"
format_stamp: "Formatted at 2026-06-20 02:00:11 on dist-test-slave-wg14"
I20260620 02:00:11.739971 27547 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260620 02:00:11.747685 27562 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:11.753722 27547 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.005s sys 0.000s
I20260620 02:00:11.755196 27547 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "2e4c0714eed64d479e0e70654e1e6a10"
format_stamp: "Formatted at 2026-06-20 02:00:11 on dist-test-slave-wg14"
I20260620 02:00:11.755541 27547 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:11.802501 27547 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:11.815407 27547 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:11.816061 27547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:11.854085 27547 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:36605
I20260620 02:00:11.854212 27613 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:36605 every 8 connection(s)
I20260620 02:00:11.857290 27547 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:00:11.858069 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 27547
I20260620 02:00:11.858289 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:00:11.864226 27614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:00:11.872058 27614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:11 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920811, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.189@KRBTEST.COM" created.
I20260620 02:00:11.914427 27614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } has no permanent_uuid. Determining permanent_uuid...
W20260620 02:00:11.916644 27565 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:44283: connect: Connection refused (error 111)
W20260620 02:00:11.920161 27614 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:44283: Network error: Client connection negotiation failed: client connection to 127.20.1.189:44283: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:00:11.962353 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.189: real 0.103s user 0.005s sys 0.025s
I20260620 02:00:11.970161 27614 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } attempt: 1
W20260620 02:00:11.973624 27614 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:44283: Network error: Client connection negotiation failed: client connection to 127.20.1.189:44283: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.189@KRBTEST.COM" created.
I20260620 02:00:12.042029 27614 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } attempt: 2
W20260620 02:00:12.045076 27614 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:44283: Network error: Client connection negotiation failed: client connection to 127.20.1.189:44283: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:00:12.051309 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:44283
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:00:12.136453 27614 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } attempt: 3
W20260620 02:00:12.141530 27614 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:44283: Network error: Client connection negotiation failed: client connection to 127.20.1.189:44283: connect: Connection refused (error 111)
I20260620 02:00:12.284935 27614 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } attempt: 4
W20260620 02:00:12.288096 27614 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:44283: Network error: Client connection negotiation failed: client connection to 127.20.1.189:44283: connect: Connection refused (error 111)
W20260620 02:00:12.553891 27623 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:12.554363 27623 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:12.554580 27623 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:12.570508 27623 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:00:12.570685 27623 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:12.570767 27623 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:00:12.570837 27623 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:00:12.587441 27614 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } attempt: 5
I20260620 02:00:12.588302 27623 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:44283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:12.590560 27623 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260620 02:00:12.591691 27614 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:44283: Network error: Client connection negotiation failed: client connection to 127.20.1.189:44283: connect: Connection refused (error 111)
I20260620 02:00:12.592759 27623 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:12.617436 27628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:12.618685 27629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:12.622193 27631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:12.623672 27623 server_base.cc:1061] running on GCE node
I20260620 02:00:12.624791 27623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:12.628671 27623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:12.632206 27623 hybrid_clock.cc:648] HybridClock initialized: now 1781920812631689 us; error 491 us; skew 500 ppm
I20260620 02:00:12.632777 27623 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:12 dist-test-slave-wg14 krb5kdc[27531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920812, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:12.638432 27623 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 02:00:12.641855 27623 webserver.cc:533] Webserver started at http://127.20.1.189:35909/ using document root <none> and password file <none>
I20260620 02:00:12.643538 27623 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:12.643770 27623 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:12.644487 27623 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:00:12.652858 27623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "ade4889be736487b89bccec2e6e336ad"
format_stamp: "Formatted at 2026-06-20 02:00:12 on dist-test-slave-wg14"
I20260620 02:00:12.654105 27623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "ade4889be736487b89bccec2e6e336ad"
format_stamp: "Formatted at 2026-06-20 02:00:12 on dist-test-slave-wg14"
I20260620 02:00:12.663686 27623 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.011s sys 0.000s
I20260620 02:00:12.670023 27638 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:12.673440 27623 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260620 02:00:12.673763 27623 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "ade4889be736487b89bccec2e6e336ad"
format_stamp: "Formatted at 2026-06-20 02:00:12 on dist-test-slave-wg14"
I20260620 02:00:12.674002 27623 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:12.697171 27623 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:12.706001 27623 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:12.706382 27623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:12.744314 27623 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:44283
I20260620 02:00:12.744351 27689 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:44283 every 8 connection(s)
I20260620 02:00:12.746531 27623 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 02:00:12.753324 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 27623
I20260620 02:00:12.753773 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 02:00:12.757006 27690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:00:12.771732 27690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.188@KRBTEST.COM" created.
Jun 20 02:00:12 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920812, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:00:12.814638 27690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:12 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920812, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:00:12.845150 27690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } has no permanent_uuid. Determining permanent_uuid...
W20260620 02:00:12.846850 27642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:36935: connect: Connection refused (error 111)
W20260620 02:00:12.849838 27690 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:36935: Network error: Client connection negotiation failed: client connection to 127.20.1.188:36935: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.188@KRBTEST.COM" created.
I20260620 02:00:12.899662 27690 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } attempt: 1
W20260620 02:00:12.903309 27690 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:36935: Network error: Client connection negotiation failed: client connection to 127.20.1.188:36935: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:00:12.938902 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:36935
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:00:12.971752 27690 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } attempt: 2
W20260620 02:00:12.974824 27690 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:36935: Network error: Client connection negotiation failed: client connection to 127.20.1.188:36935: connect: Connection refused (error 111)
I20260620 02:00:13.066100 27690 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } attempt: 3
W20260620 02:00:13.069033 27690 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:36935: Network error: Client connection negotiation failed: client connection to 127.20.1.188:36935: connect: Connection refused (error 111)
I20260620 02:00:13.139231 27614 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } attempt: 6
Jun 20 02:00:13 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920811, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:00:13.161872 27614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } has no permanent_uuid. Determining permanent_uuid...
W20260620 02:00:13.165704 27614 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:36935: Network error: Client connection negotiation failed: client connection to 127.20.1.188:36935: connect: Connection refused (error 111)
I20260620 02:00:13.212293 27690 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } attempt: 4
W20260620 02:00:13.214778 27690 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:36935: Network error: Client connection negotiation failed: client connection to 127.20.1.188:36935: connect: Connection refused (error 111)
I20260620 02:00:13.218530 27614 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } attempt: 1
W20260620 02:00:13.222715 27614 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:36935: Network error: Client connection negotiation failed: client connection to 127.20.1.188:36935: connect: Connection refused (error 111)
I20260620 02:00:13.297062 27614 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } attempt: 2
W20260620 02:00:13.299988 27614 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:36935: Network error: Client connection negotiation failed: client connection to 127.20.1.188:36935: connect: Connection refused (error 111)
I20260620 02:00:13.413349 27614 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } attempt: 3
W20260620 02:00:13.415416 27614 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:36935: Network error: Client connection negotiation failed: client connection to 127.20.1.188:36935: connect: Connection refused (error 111)
W20260620 02:00:13.502045 27700 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:13.502496 27700 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:13.502666 27700 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:13.513751 27700 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:00:13.513914 27700 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:13.513994 27700 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:00:13.514055 27700 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:00:13.515542 27690 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } attempt: 5
W20260620 02:00:13.518975 27690 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:36935: Network error: Client connection negotiation failed: client connection to 127.20.1.188:36935: connect: Connection refused (error 111)
I20260620 02:00:13.532943 27700 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:36935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:13.535041 27700 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:13.538451 27700 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:13.556629 27707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:13.557638 27708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:13.560421 27710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:13.562572 27700 server_base.cc:1061] running on GCE node
I20260620 02:00:13.563725 27700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:13.564740 27614 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } attempt: 4
I20260620 02:00:13.566636 27700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260620 02:00:13.567392 27614 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:36935: Network error: Client connection negotiation failed: client connection to 127.20.1.188:36935: connect: Connection refused (error 111)
I20260620 02:00:13.568213 27700 hybrid_clock.cc:648] HybridClock initialized: now 1781920813568106 us; error 74 us; skew 500 ppm
I20260620 02:00:13.568660 27700 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:13 dist-test-slave-wg14 krb5kdc[27531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920813, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:13.573477 27700 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 02:00:13.577210 27700 webserver.cc:533] Webserver started at http://127.20.1.188:40725/ using document root <none> and password file <none>
I20260620 02:00:13.578397 27700 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:13.578541 27700 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:13.579334 27700 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:00:13.583810 27700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "aad24f3b73c441988ddb1b60cf335584"
format_stamp: "Formatted at 2026-06-20 02:00:13 on dist-test-slave-wg14"
I20260620 02:00:13.585222 27700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "aad24f3b73c441988ddb1b60cf335584"
format_stamp: "Formatted at 2026-06-20 02:00:13 on dist-test-slave-wg14"
I20260620 02:00:13.593550 27700 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s
I20260620 02:00:13.599931 27717 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:13.604393 27700 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.005s sys 0.001s
I20260620 02:00:13.604677 27700 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "aad24f3b73c441988ddb1b60cf335584"
format_stamp: "Formatted at 2026-06-20 02:00:13 on dist-test-slave-wg14"
I20260620 02:00:13.605085 27700 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:13.621682 27700 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:13.630270 27700 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:13.630604 27700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:13.670089 27700 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:36935
I20260620 02:00:13.670135 27768 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:36935 every 8 connection(s)
I20260620 02:00:13.672168 27700 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 02:00:13.680266 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 27700
I20260620 02:00:13.680475 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 02:00:13.681080 27769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:00:13.691603 27769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:13 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 02:00:13 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920813, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 02:00:13 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:00:13.727454 27769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:13 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920813, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:00:13.753206 27769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:13 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 02:00:13 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920813, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:00:13.792241 27769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584: Bootstrap starting.
I20260620 02:00:13.799284 27769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584: Neither blocks nor log segments found. Creating new log.
I20260620 02:00:13.800812 27769 log.cc:826] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584: Log is configured to *not* fsync() on all Append() calls
I20260620 02:00:13.806571 27769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584: No bootstrap required, opened a new log
I20260620 02:00:13.814796 27769 raft_consensus.cc:359] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } }
I20260620 02:00:13.815478 27769 raft_consensus.cc:385] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:00:13.815670 27769 raft_consensus.cc:740] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad24f3b73c441988ddb1b60cf335584, State: Initialized, Role: FOLLOWER
I20260620 02:00:13.816540 27769 consensus_queue.cc:260] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } }
I20260620 02:00:13.817757 27780 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } }
I20260620 02:00:13.818259 27780 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:13.819682 27769 sys_catalog.cc:565] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:00:13.835660 27614 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } attempt: 5
I20260620 02:00:13.831874 27700 master_runner.cc:187] Error getting master registration for 127.20.1.190:36605: OK, instance_id { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" instance_seqno: 1781920811841597 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:00:13.842515 27700 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Jun 20 02:00:13 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920811, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
W20260620 02:00:13.850857 27791 catalog_manager.cc:1593] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:00:13.851087 27791 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:00:13.866379 27614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10: Bootstrap starting.
I20260620 02:00:13.873759 27614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10: Neither blocks nor log segments found. Creating new log.
I20260620 02:00:13.875341 27614 log.cc:826] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10: Log is configured to *not* fsync() on all Append() calls
I20260620 02:00:13.879338 27614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10: No bootstrap required, opened a new log
W20260620 02:00:13.884054 27723 tablet.cc:2406] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:00:13.885977 27614 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } }
I20260620 02:00:13.886608 27614 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:00:13.886948 27614 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4c0714eed64d479e0e70654e1e6a10, State: Initialized, Role: FOLLOWER
I20260620 02:00:13.888484 27614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } }
I20260620 02:00:13.894522 27794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } }
I20260620 02:00:13.896262 27794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:13.896984 27614 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:00:13.921207 27805 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:00:13.921424 27805 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:00:13.914466 27547 master_runner.cc:187] Error getting master registration for 127.20.1.189:44283: OK, instance_id { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" instance_seqno: 1781920812736820 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:00:13.924381 27547 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:00:14.067169 27690 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } attempt: 6
W20260620 02:00:14.074707 27568 tablet.cc:2406] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 02:00:14 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920812, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:00:14.128806 27690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad: Bootstrap starting.
I20260620 02:00:14.136406 27690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad: Neither blocks nor log segments found. Creating new log.
I20260620 02:00:14.138285 27690 log.cc:826] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad: Log is configured to *not* fsync() on all Append() calls
I20260620 02:00:14.144963 27690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad: No bootstrap required, opened a new log
I20260620 02:00:14.154836 27690 raft_consensus.cc:359] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } }
I20260620 02:00:14.155360 27690 raft_consensus.cc:385] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:00:14.155516 27690 raft_consensus.cc:740] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade4889be736487b89bccec2e6e336ad, State: Initialized, Role: FOLLOWER
I20260620 02:00:14.156467 27690 consensus_queue.cc:260] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } }
I20260620 02:00:14.159986 27809 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } }
I20260620 02:00:14.160588 27809 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:14.164947 27690 sys_catalog.cc:565] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:00:14.212327 27644 tablet.cc:2406] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:00:14.225895 27623 master_runner.cc:264] No leader master found from master ade4889be736487b89bccec2e6e336ad
I20260620 02:00:14.226570 27623 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 02:00:14.242313 27820 catalog_manager.cc:1593] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:00:14.242574 27820 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:00:14.256242 27780 raft_consensus.cc:493] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:00:14.256734 27780 raft_consensus.cc:515] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } }
I20260620 02:00:14.263684 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aad24f3b73c441988ddb1b60cf335584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4c0714eed64d479e0e70654e1e6a10" is_pre_election: true
I20260620 02:00:14.265386 27589 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad24f3b73c441988ddb1b60cf335584 in term 0.
I20260620 02:00:14.265645 27794 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:00:14.265820 27794 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } }
I20260620 02:00:14.270025 27665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e4c0714eed64d479e0e70654e1e6a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade4889be736487b89bccec2e6e336ad" is_pre_election: true
I20260620 02:00:14.276876 27744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e4c0714eed64d479e0e70654e1e6a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad24f3b73c441988ddb1b60cf335584" is_pre_election: true
I20260620 02:00:14.277586 27744 raft_consensus.cc:2468] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e4c0714eed64d479e0e70654e1e6a10 in term 0.
I20260620 02:00:14.279351 27566 leader_election.cc:304] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e4c0714eed64d479e0e70654e1e6a10, aad24f3b73c441988ddb1b60cf335584; no voters:
I20260620 02:00:14.281874 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aad24f3b73c441988ddb1b60cf335584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade4889be736487b89bccec2e6e336ad" is_pre_election: true
I20260620 02:00:14.282223 27822 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:00:14.282415 27822 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:00:14.282563 27822 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:00:14.282613 27664 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad24f3b73c441988ddb1b60cf335584 in term 0.
I20260620 02:00:14.284837 27720 leader_election.cc:304] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad24f3b73c441988ddb1b60cf335584, ade4889be736487b89bccec2e6e336ad; no voters:
I20260620 02:00:14.287151 27794 leader_election.cc:290] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ade4889be736487b89bccec2e6e336ad (127.20.1.189:44283), aad24f3b73c441988ddb1b60cf335584 (127.20.1.188:36935)
I20260620 02:00:14.287509 27822 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } }
I20260620 02:00:14.288650 27780 leader_election.cc:290] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e4c0714eed64d479e0e70654e1e6a10 (127.20.1.190:36605), ade4889be736487b89bccec2e6e336ad (127.20.1.189:44283)
I20260620 02:00:14.290954 27780 raft_consensus.cc:2804] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:00:14.292326 27665 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e4c0714eed64d479e0e70654e1e6a10 in term 0.
I20260620 02:00:14.297405 27780 raft_consensus.cc:493] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:00:14.297924 27780 raft_consensus.cc:3060] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:00:14.297137 27744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e4c0714eed64d479e0e70654e1e6a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad24f3b73c441988ddb1b60cf335584"
I20260620 02:00:14.292430 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e4c0714eed64d479e0e70654e1e6a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade4889be736487b89bccec2e6e336ad"
I20260620 02:00:14.300523 27664 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:00:14.307531 27664 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e4c0714eed64d479e0e70654e1e6a10 in term 1.
I20260620 02:00:14.313688 27822 leader_election.cc:290] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [CANDIDATE]: Term 1 election: Requested vote from peers ade4889be736487b89bccec2e6e336ad (127.20.1.189:44283), aad24f3b73c441988ddb1b60cf335584 (127.20.1.188:36935)
I20260620 02:00:14.314505 27565 leader_election.cc:304] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e4c0714eed64d479e0e70654e1e6a10, ade4889be736487b89bccec2e6e336ad; no voters:
I20260620 02:00:14.315254 27822 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:00:14.315943 27822 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 1 LEADER]: Becoming Leader. State: Replica: 2e4c0714eed64d479e0e70654e1e6a10, State: Running, Role: LEADER
I20260620 02:00:14.315800 27780 raft_consensus.cc:515] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } }
I20260620 02:00:14.316629 27744 raft_consensus.cc:2393] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2e4c0714eed64d479e0e70654e1e6a10 in current term 1: Already voted for candidate aad24f3b73c441988ddb1b60cf335584 in this term.
I20260620 02:00:14.317754 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aad24f3b73c441988ddb1b60cf335584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4c0714eed64d479e0e70654e1e6a10"
I20260620 02:00:14.318120 27780 leader_election.cc:290] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [CANDIDATE]: Term 1 election: Requested vote from peers 2e4c0714eed64d479e0e70654e1e6a10 (127.20.1.190:36605), ade4889be736487b89bccec2e6e336ad (127.20.1.189:44283)
I20260620 02:00:14.323755 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aad24f3b73c441988ddb1b60cf335584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade4889be736487b89bccec2e6e336ad"
I20260620 02:00:14.324287 27664 raft_consensus.cc:2393] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aad24f3b73c441988ddb1b60cf335584 in current term 1: Already voted for candidate 2e4c0714eed64d479e0e70654e1e6a10 in this term.
I20260620 02:00:14.326467 27720 leader_election.cc:304] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aad24f3b73c441988ddb1b60cf335584; no voters: 2e4c0714eed64d479e0e70654e1e6a10, ade4889be736487b89bccec2e6e336ad
I20260620 02:00:14.335309 27780 raft_consensus.cc:2749] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 02:00:14.336293 27822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } }
I20260620 02:00:14.369949 27794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e4c0714eed64d479e0e70654e1e6a10. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } }
I20260620 02:00:14.370703 27794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:14.376204 27825 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:00:14.395921 27825 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:00:14.441324 27744 raft_consensus.cc:1275] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 1 FOLLOWER]: Refusing update from remote peer 2e4c0714eed64d479e0e70654e1e6a10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:00:14.442665 27794 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:00:14.466347 27664 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 1 FOLLOWER]: Refusing update from remote peer 2e4c0714eed64d479e0e70654e1e6a10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:00:14.475802 27794 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260620 02:00:14.503314 27780 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e4c0714eed64d479e0e70654e1e6a10. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } }
I20260620 02:00:14.503716 27780 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:14.524940 27809 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e4c0714eed64d479e0e70654e1e6a10. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } }
I20260620 02:00:14.525431 27809 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:14.530737 27827 mvcc.cc:204] Tried to move back new op lower bound from 7298747655937880064 to 7298747655545868288. Current Snapshot: MvccSnapshot[applied={T|T < 7298747655937880064}]
I20260620 02:00:14.538966 27822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } }
I20260620 02:00:14.540643 27822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:14.541091 27822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } }
I20260620 02:00:14.541344 27822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:14.542465 27825 catalog_manager.cc:1382] Generated new cluster ID: a06cac6b748c44a980d6931aeb6f244c
I20260620 02:00:14.542595 27825 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:00:14.561729 27809 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } }
I20260620 02:00:14.562031 27809 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:14.577404 27780 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } }
I20260620 02:00:14.577891 27780 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:14.683164 27825 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:00:14.688632 27825 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:00:14.819150 27825 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10: Generated new TSK 0
I20260620 02:00:14.821982 27825 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 02:00:14.853781 27791 catalog_manager.cc:1294] Loaded cluster ID: a06cac6b748c44a980d6931aeb6f244c
I20260620 02:00:14.853929 27791 catalog_manager.cc:1587] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584: loading cluster ID for follower catalog manager: success
I20260620 02:00:14.875764 27791 catalog_manager.cc:1609] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584: acquiring CA information for follower catalog manager: success
I20260620 02:00:14.881538 27791 catalog_manager.cc:1637] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:00:15.000384 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.138s user 0.000s sys 0.034s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:00:15.101480 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:00:15.244895 27820 catalog_manager.cc:1294] Loaded cluster ID: a06cac6b748c44a980d6931aeb6f244c
I20260620 02:00:15.245028 27820 catalog_manager.cc:1587] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad: loading cluster ID for follower catalog manager: success
I20260620 02:00:15.248478 27820 catalog_manager.cc:1609] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad: acquiring CA information for follower catalog manager: success
I20260620 02:00:15.250953 27820 catalog_manager.cc:1637] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 02:00:15.585788 27839 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:15.586215 27839 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:15.586357 27839 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:15.596197 27839 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:15.596465 27839 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:00:15.612329 27839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:15.614832 27839 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:15.618564 27839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:15.636523 27844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:15.637125 27845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:15.637351 27847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:15.638387 27839 server_base.cc:1061] running on GCE node
I20260620 02:00:15.639223 27839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:15.640460 27839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:15.642199 27839 hybrid_clock.cc:648] HybridClock initialized: now 1781920815642111 us; error 76 us; skew 500 ppm
I20260620 02:00:15.642889 27839 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:15 dist-test-slave-wg14 krb5kdc[27531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920815, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:15.648442 27839 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:00:15.650611 27839 webserver.cc:533] Webserver started at http://127.20.1.129:33517/ using document root <none> and password file <none>
I20260620 02:00:15.651664 27839 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:15.651834 27839 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:15.652223 27839 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:00:15.655211 27839 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "e27cbc9392f244b0a428f83525772354"
format_stamp: "Formatted at 2026-06-20 02:00:15 on dist-test-slave-wg14"
I20260620 02:00:15.656684 27839 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "e27cbc9392f244b0a428f83525772354"
format_stamp: "Formatted at 2026-06-20 02:00:15 on dist-test-slave-wg14"
I20260620 02:00:15.668092 27839 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.007s sys 0.005s
I20260620 02:00:15.673029 27854 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:15.675290 27839 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:00:15.675647 27839 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "e27cbc9392f244b0a428f83525772354"
format_stamp: "Formatted at 2026-06-20 02:00:15 on dist-test-slave-wg14"
I20260620 02:00:15.675967 27839 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:15.698004 27839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:15.705382 27839 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:15.705904 27839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:15.707370 27839 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:00:15.712770 27839 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:00:15.712932 27839 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:15.713056 27839 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:00:15.713124 27839 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:15.767913 27839 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:43833
I20260620 02:00:15.767968 27966 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:43833 every 8 connection(s)
I20260620 02:00:15.769733 27839 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:00:15.771971 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 27839
I20260620 02:00:15.772231 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 02:00:15 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920815, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 02:00:15 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920815, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 02:00:15 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920815, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:00:15.825814 27968 heartbeater.cc:344] Connected to a master server at 127.20.1.190:36605
I20260620 02:00:15.826297 27968 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:15.827618 27968 heartbeater.cc:507] Master 127.20.1.190:36605 requested a full tablet report, sending...
I20260620 02:00:15.828890 27967 heartbeater.cc:344] Connected to a master server at 127.20.1.188:36935
I20260620 02:00:15.829121 27967 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:15.829718 27967 heartbeater.cc:507] Master 127.20.1.188:36935 requested a full tablet report, sending...
I20260620 02:00:15.830088 27969 heartbeater.cc:344] Connected to a master server at 127.20.1.189:44283
I20260620 02:00:15.830274 27969 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:15.830839 27969 heartbeater.cc:507] Master 127.20.1.189:44283 requested a full tablet report, sending...
I20260620 02:00:15.831065 27579 ts_manager.cc:194] Registered new tserver with Master: e27cbc9392f244b0a428f83525772354 (127.20.1.129:43833)
I20260620 02:00:15.831735 27734 ts_manager.cc:194] Registered new tserver with Master: e27cbc9392f244b0a428f83525772354 (127.20.1.129:43833)
I20260620 02:00:15.832741 27654 ts_manager.cc:194] Registered new tserver with Master: e27cbc9392f244b0a428f83525772354 (127.20.1.129:43833)
I20260620 02:00:15.833427 27579 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:34921
I20260620 02:00:15.855239 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
WARNING: no policy specified for kudu/127.20.1.187@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.187@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.187 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.187 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.187@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.187@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.187 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.187 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:00:16.048553 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.187: real 0.106s user 0.003s sys 0.015s
I20260620 02:00:16.049502 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:37251
--webserver_interface=127.20.1.187
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.187
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935,127.20.1.187:37251 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:00:16.699265 27981 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:16.699733 27981 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:16.699885 27981 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:16.716897 27981 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:00:16.717059 27981 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:16.717128 27981 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:00:16.717217 27981 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:00:16.731724 27981 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935,127.20.1.187:37251
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.187
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:37251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.20.1.187
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:16.735689 27981 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:16.738000 27981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:16.751722 27986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:16.756748 27987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:16.759083 27989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:16.759299 27981 server_base.cc:1061] running on GCE node
I20260620 02:00:16.760392 27981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:16.762669 27981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:16.766255 27981 hybrid_clock.cc:648] HybridClock initialized: now 1781920816766088 us; error 135 us; skew 500 ppm
I20260620 02:00:16.766804 27981 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:16 dist-test-slave-wg14 krb5kdc[27531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920816, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:16.772658 27981 init.cc:377] Logged in from keytab as kudu/127.20.1.187@KRBTEST.COM (short username kudu)
I20260620 02:00:16.776486 27981 webserver.cc:533] Webserver started at http://127.20.1.187:41713/ using document root <none> and password file <none>
I20260620 02:00:16.777631 27981 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:16.777834 27981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:16.778517 27981 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:00:16.781751 27981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/data/instance:
uuid: "040c6f27fd3f4093b2d18cd433480f6c"
format_stamp: "Formatted at 2026-06-20 02:00:16 on dist-test-slave-wg14"
I20260620 02:00:16.782908 27981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/wal/instance:
uuid: "040c6f27fd3f4093b2d18cd433480f6c"
format_stamp: "Formatted at 2026-06-20 02:00:16 on dist-test-slave-wg14"
I20260620 02:00:16.793433 27981 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.007s sys 0.000s
I20260620 02:00:16.800086 27996 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:16.802237 27981 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:00:16.802490 27981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "040c6f27fd3f4093b2d18cd433480f6c"
format_stamp: "Formatted at 2026-06-20 02:00:16 on dist-test-slave-wg14"
I20260620 02:00:16.802739 27981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:16.836405 27981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:16.837669 27968 heartbeater.cc:499] Master 127.20.1.190:36605 was elected leader, sending a full tablet report...
I20260620 02:00:16.847280 27981 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:16.848945 27981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:16.885135 27981 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:37251
I20260620 02:00:16.885282 28047 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:37251 every 8 connection(s)
I20260620 02:00:16.888417 27981 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 02:00:16.893553 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 27981
I20260620 02:00:16.901691 28048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:00:16.917069 28048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:16 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
Jun 20 02:00:16 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920816, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:00:16.946295 28048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:16 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920816, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:00:16.971680 28048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:16 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920816, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:00:16.999367 28048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:17 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920816, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
I20260620 02:00:17.029268 28048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: Bootstrap starting.
I20260620 02:00:17.035055 28048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: Neither blocks nor log segments found. Creating new log.
I20260620 02:00:17.037912 28048 log.cc:826] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:00:17.043370 28048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: No bootstrap required, opened a new log
I20260620 02:00:17.050817 28048 raft_consensus.cc:359] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } }
I20260620 02:00:17.051492 28048 raft_consensus.cc:385] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:00:17.051631 28048 raft_consensus.cc:740] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040c6f27fd3f4093b2d18cd433480f6c, State: Initialized, Role: FOLLOWER
I20260620 02:00:17.052371 28048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } }
I20260620 02:00:17.053478 28057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } } }
I20260620 02:00:17.053809 28057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:17.054862 28048 sys_catalog.cc:565] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:00:17.071391 28068 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:00:17.071691 28068 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:00:17.086922 27981 master_runner.cc:429] Detected that this master 040c6f27fd3f4093b2d18cd433480f6c is joining an existing cluster
I20260620 02:00:17.087697 27981 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260620 02:00:17.094781 27981 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 02:00:17.095854 27981 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 02:00:17.096316 27981 tablet_replica.cc:333] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: stopping tablet replica
I20260620 02:00:17.099732 27981 master_runner.cc:434] Initiating AddMaster RPC to add 127.20.1.187:37251
Jun 20 02:00:17 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920811, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
I20260620 02:00:17.214073 27579 catalog_manager.cc:7201] Initiating ChangeConfig request to add master 127.20.1.187:37251: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c"
member_type: NON_VOTER
last_known_addr {
host: "127.20.1.187"
port: 37251
}
attrs {
promote: true
}
}
dest_uuid: "2e4c0714eed64d479e0e70654e1e6a10"
cas_config_opid_index: -1
I20260620 02:00:17.216248 27579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: true } }
I20260620 02:00:17.220927 27664 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 1 FOLLOWER]: Refusing update from remote peer 2e4c0714eed64d479e0e70654e1e6a10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 02:00:17.223090 27834 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260620 02:00:17.223987 27564 consensus_peers.cc:597] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 -> Peer 040c6f27fd3f4093b2d18cd433480f6c (127.20.1.187:37251): Couldn't send request to peer 040c6f27fd3f4093b2d18cd433480f6c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260620 02:00:17.225715 27743 raft_consensus.cc:1275] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 1 FOLLOWER]: Refusing update from remote peer 2e4c0714eed64d479e0e70654e1e6a10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 02:00:17.230355 28080 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 02:00:17.231973 28080 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 040c6f27fd3f4093b2d18cd433480f6c (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: true } } }
I20260620 02:00:17.232968 27664 raft_consensus.cc:2955] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 040c6f27fd3f4093b2d18cd433480f6c (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: true } } }
I20260620 02:00:17.239189 28080 catalog_manager.cc:7205] Successfully completed master ChangeConfig request to add master 127.20.1.187:37251
I20260620 02:00:17.241340 27981 master.cc:562] Master@127.20.1.187:37251 shutting down...
I20260620 02:00:17.247510 28083 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: true } } }
I20260620 02:00:17.247915 28083 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:17.258298 27834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: true } } }
I20260620 02:00:17.259054 27834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:17.261186 27743 raft_consensus.cc:2955] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 040c6f27fd3f4093b2d18cd433480f6c (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: true } } }
I20260620 02:00:17.264357 28084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: true } } }
I20260620 02:00:17.264798 28084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:17.268050 28084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: true } } }
I20260620 02:00:17.270623 28084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: This master's current role is: LEADER
W20260620 02:00:17.276625 27529 connection.cc:573] client connection to 127.20.1.187:37251 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:00:17.285638 27981 master.cc:584] Master@127.20.1.187:37251 shutdown complete.
I20260620 02:00:17.290611 27981 master_runner.cc:306] Clearing existing system tablet
I20260620 02:00:17.291483 27981 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:17.291662 27981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:17.296180 28087 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: true } } }
I20260620 02:00:17.296734 28087 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:17.297971 27981 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:00:17.302340 28089 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:17.303056 27981 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:17.303711 27981 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:00:17.303843 27981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "040c6f27fd3f4093b2d18cd433480f6c"
format_stamp: "Formatted at 2026-06-20 02:00:16 on dist-test-slave-wg14"
I20260620 02:00:17.309334 27981 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:00:17.323498 27981 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 02:00:17.323941 27981 log.cc:1199] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: Deleting WAL directory at /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260620 02:00:17.324551 27981 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: Deleting consensus metadata
I20260620 02:00:17.324849 27981 master_runner.cc:316] Copying system tablet from 127.20.1.190:36605
I20260620 02:00:17.337360 27981 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.1.190:36605
I20260620 02:00:17.364831 27599 tablet_copy_service.cc:140] P 2e4c0714eed64d479e0e70654e1e6a10: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 040c6f27fd3f4093b2d18cd433480f6c ({username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:43464)
I20260620 02:00:17.365110 27599 tablet_copy_service.cc:161] P 2e4c0714eed64d479e0e70654e1e6a10: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 040c6f27fd3f4093b2d18cd433480f6c at {username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:43464: session id = 040c6f27fd3f4093b2d18cd433480f6c-00000000000000000000000000000000
I20260620 02:00:17.369629 27599 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 02:00:17.376834 27981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 02:00:17.385205 27981 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: tablet copy: Starting download of 0 data blocks...
I20260620 02:00:17.385684 27981 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: tablet copy: Starting download of 1 WAL segments...
I20260620 02:00:17.392870 27981 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 02:00:17.400640 27599 tablet_copy_service.cc:342] P 2e4c0714eed64d479e0e70654e1e6a10: Request end of tablet copy session 040c6f27fd3f4093b2d18cd433480f6c-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:43464
I20260620 02:00:17.401183 27599 tablet_copy_service.cc:434] P 2e4c0714eed64d479e0e70654e1e6a10: ending tablet copy session 040c6f27fd3f4093b2d18cd433480f6c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 040c6f27fd3f4093b2d18cd433480f6c
W20260620 02:00:17.416410 27981 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260620 02:00:17.421368 27981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:17.426522 28096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:17.427543 28097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:17.439810 27981 server_base.cc:1061] running on GCE node
W20260620 02:00:17.439752 28099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:17.440471 27981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:17.441550 27981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:17.442816 27981 hybrid_clock.cc:648] HybridClock initialized: now 1781920817442762 us; error 44 us; skew 500 ppm
I20260620 02:00:17.443234 27981 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:17 dist-test-slave-wg14 krb5kdc[27531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920817, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:17.446622 27981 init.cc:377] Logged in from keytab as kudu/127.20.1.187@KRBTEST.COM (short username kudu)
I20260620 02:00:17.450683 27981 webserver.cc:533] Webserver started at http://127.20.1.187:41097/ using document root <none> and password file <none>
I20260620 02:00:17.451241 27981 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:17.451355 27981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:17.455824 27981 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:00:17.461673 28106 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:17.462898 27981 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 02:00:17.463215 27981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "040c6f27fd3f4093b2d18cd433480f6c"
format_stamp: "Formatted at 2026-06-20 02:00:16 on dist-test-slave-wg14"
I20260620 02:00:17.463502 27981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:17.487557 27981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:17.497476 27981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:17.534530 27981 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:37251
I20260620 02:00:17.535354 28157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:37251 every 8 connection(s)
I20260620 02:00:17.536111 27981 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 02:00:17.543452 28159 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:00:17.548413 28159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: Bootstrap starting.
I20260620 02:00:17.572053 28159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:00:17.572624 28159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: Bootstrap complete.
I20260620 02:00:17.573858 28159 raft_consensus.cc:359] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: true } }
I20260620 02:00:17.574335 28159 raft_consensus.cc:740] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 040c6f27fd3f4093b2d18cd433480f6c, State: Initialized, Role: LEARNER
I20260620 02:00:17.574628 28159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: true } }
I20260620 02:00:17.575660 28162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: true } } }
I20260620 02:00:17.576431 28162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: This master's current role is: LEARNER
I20260620 02:00:17.577047 28159 sys_catalog.cc:565] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:00:17.600392 28173 catalog_manager.cc:1294] Loaded cluster ID: a06cac6b748c44a980d6931aeb6f244c
I20260620 02:00:17.600504 28173 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: loading cluster ID for follower catalog manager: success
I20260620 02:00:17.606462 28173 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: acquiring CA information for follower catalog manager: success
I20260620 02:00:17.617794 28173 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:00:17.618207 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 27839
Jun 20 02:00:17 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:00:17.637553 28133 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260620 02:00:17.638682 28162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e4c0714eed64d479e0e70654e1e6a10. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: true } } }
I20260620 02:00:17.639094 28162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: This master's current role is: LEARNER
I20260620 02:00:17.653745 27962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:00:17.789049 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 27839
I20260620 02:00:17.836828 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:43833
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935
--webserver_port=33517
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935,127.20.1.187:37251 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:00:18.052629 27834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: true } } }
I20260620 02:00:18.053042 27834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:18.055207 28178 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10: attempting to promote NON_VOTER 040c6f27fd3f4093b2d18cd433480f6c to VOTER
I20260620 02:00:18.058586 28178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } }
I20260620 02:00:18.062507 27743 raft_consensus.cc:1275] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 1 FOLLOWER]: Refusing update from remote peer 2e4c0714eed64d479e0e70654e1e6a10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:00:18.062608 28133 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 1 LEARNER]: Refusing update from remote peer 2e4c0714eed64d479e0e70654e1e6a10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:00:18.063294 27664 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 1 FOLLOWER]: Refusing update from remote peer 2e4c0714eed64d479e0e70654e1e6a10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:00:18.063439 28178 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:00:18.064170 27834 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:00:18.064606 28177 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:00:18.068799 28177 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 040c6f27fd3f4093b2d18cd433480f6c (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:18.069967 27664 raft_consensus.cc:2955] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 040c6f27fd3f4093b2d18cd433480f6c (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:18.074745 27834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:18.075313 27834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:18.075937 28177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:18.076200 27834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:18.076478 28177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:18.076658 27743 raft_consensus.cc:2955] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 040c6f27fd3f4093b2d18cd433480f6c (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:18.077135 27834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:18.084148 28181 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:18.086961 28132 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 040c6f27fd3f4093b2d18cd433480f6c (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:18.088217 28181 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:18.085271 28182 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:18.091835 28182 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:18.092203 28177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:18.092590 28177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:18.093721 28162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e4c0714eed64d479e0e70654e1e6a10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:18.094043 28162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: This master's current role is: FOLLOWER
W20260620 02:00:18.369292 28176 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:18.369870 28176 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:18.370198 28176 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:18.384740 28176 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:18.385028 28176 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:00:18.399324 28176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:43833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=33517
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935,127.20.1.187:37251
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:18.402205 28176 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:18.406787 28176 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:18.423416 28189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:18.426196 28190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:18.427507 28192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:18.427999 28176 server_base.cc:1061] running on GCE node
I20260620 02:00:18.428740 28176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:18.429988 28176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:18.431300 28176 hybrid_clock.cc:648] HybridClock initialized: now 1781920818431209 us; error 76 us; skew 500 ppm
I20260620 02:00:18.431733 28176 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:18 dist-test-slave-wg14 krb5kdc[27531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920818, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:18.436467 28176 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:00:18.439455 28176 webserver.cc:533] Webserver started at http://127.20.1.129:33517/ using document root <none> and password file <none>
I20260620 02:00:18.440366 28176 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:18.440524 28176 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:18.453711 28176 fs_manager.cc:714] Time spent opening directory manager: real 0.011s user 0.005s sys 0.003s
I20260620 02:00:18.460368 28199 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:18.463581 28176 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260620 02:00:18.463896 28176 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "e27cbc9392f244b0a428f83525772354"
format_stamp: "Formatted at 2026-06-20 02:00:15 on dist-test-slave-wg14"
I20260620 02:00:18.464931 28176 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:18.488541 28176 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:18.496138 28176 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:18.496490 28176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:18.498709 28176 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:00:18.502311 28176 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:00:18.502462 28176 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:18.502668 28176 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:00:18.502761 28176 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:18.560806 28176 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:43833
I20260620 02:00:18.560894 28311 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:43833 every 8 connection(s)
I20260620 02:00:18.564664 28176 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:00:18.568691 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 28176
Jun 20 02:00:18 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920818, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
Jun 20 02:00:18 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920818, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 02:00:18 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920818, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 02:00:18 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920818, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:00:18.630393 28312 heartbeater.cc:344] Connected to a master server at 127.20.1.187:37251
I20260620 02:00:18.630944 28312 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:18.632403 28312 heartbeater.cc:507] Master 127.20.1.187:37251 requested a full tablet report, sending...
I20260620 02:00:18.634563 28313 heartbeater.cc:344] Connected to a master server at 127.20.1.188:36935
I20260620 02:00:18.634760 28313 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:18.635569 28313 heartbeater.cc:507] Master 127.20.1.188:36935 requested a full tablet report, sending...
I20260620 02:00:18.635773 28123 ts_manager.cc:194] Registered new tserver with Master: e27cbc9392f244b0a428f83525772354 (127.20.1.129:43833)
I20260620 02:00:18.641500 27734 ts_manager.cc:194] Re-registered known tserver with Master: e27cbc9392f244b0a428f83525772354 (127.20.1.129:43833)
I20260620 02:00:18.643734 28314 heartbeater.cc:344] Connected to a master server at 127.20.1.190:36605
I20260620 02:00:18.643954 28314 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:18.644681 28314 heartbeater.cc:507] Master 127.20.1.190:36605 requested a full tablet report, sending...
I20260620 02:00:18.647739 27579 ts_manager.cc:194] Re-registered known tserver with Master: e27cbc9392f244b0a428f83525772354 (127.20.1.129:43833)
I20260620 02:00:18.648268 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 02:00:18.649158 27579 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:48943
I20260620 02:00:18.649320 28315 heartbeater.cc:344] Connected to a master server at 127.20.1.189:44283
I20260620 02:00:18.649534 28315 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:18.650223 28315 heartbeater.cc:507] Master 127.20.1.189:44283 requested a full tablet report, sending...
I20260620 02:00:18.652113 27654 ts_manager.cc:194] Re-registered known tserver with Master: e27cbc9392f244b0a428f83525772354 (127.20.1.129:43833)
I20260620 02:00:18.666567 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 02:00:18.666798 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 27547
I20260620 02:00:18.668587 27609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:00:18.825816 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 27547
W20260620 02:00:18.854303 27720 connection.cc:573] client connection to 127.20.1.190:36605 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 02:00:18.854365 27639 connection.cc:573] server connection from 127.0.0.1:33718 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:00:18.854832 27700 master_runner.cc:187] Error getting master registration for 127.20.1.190:36605: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107),
I20260620 02:00:18.855114 27700 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:00:18.855247 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:36605
--webserver_interface=127.20.1.190
--webserver_port=34087
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935
--master_addresses=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935,127.20.1.187:37251 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:00:19.232259 27623 master_runner.cc:187] Error getting master registration for 127.20.1.190:36605: Network error: Client connection negotiation failed: client connection to 127.20.1.190:36605: connect: Connection refused (error 111),
I20260620 02:00:19.237349 27623 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 02:00:19.378080 28326 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:19.378635 28326 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:19.378901 28326 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:19.389724 28326 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:00:19.389858 28326 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:19.389914 28326 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:00:19.389954 28326 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:00:19.404385 28326 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935,127.20.1.187:37251
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:36605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=34087
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:19.406595 28326 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:19.408550 28326 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:19.420070 28333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:19.420377 28332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:19.420632 28335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:19.422957 28326 server_base.cc:1061] running on GCE node
I20260620 02:00:19.425076 28326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:19.427641 28326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:19.431337 28326 hybrid_clock.cc:648] HybridClock initialized: now 1781920819431204 us; error 109 us; skew 500 ppm
I20260620 02:00:19.431907 28326 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:19 dist-test-slave-wg14 krb5kdc[27531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920819, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:19.436319 28326 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:00:19.438503 28326 webserver.cc:533] Webserver started at http://127.20.1.190:34087/ using document root <none> and password file <none>
I20260620 02:00:19.439724 28326 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:19.439915 28326 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:19.448905 28326 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 02:00:19.456368 28342 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:19.458354 28326 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.005s sys 0.000s
I20260620 02:00:19.458600 28326 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "2e4c0714eed64d479e0e70654e1e6a10"
format_stamp: "Formatted at 2026-06-20 02:00:11 on dist-test-slave-wg14"
I20260620 02:00:19.459589 28326 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:19.509554 28326 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:19.521297 28326 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:19.521625 28326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:19.566548 28326 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:36605
I20260620 02:00:19.566577 28393 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:36605 every 8 connection(s)
I20260620 02:00:19.572115 28326 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:00:19.580221 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 28326
I20260620 02:00:19.580549 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 28176
I20260620 02:00:19.587921 28394 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:00:19.592212 28394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10: Bootstrap starting.
I20260620 02:00:19.618736 28307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:00:19.622014 28394 log.cc:826] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10: Log is configured to *not* fsync() on all Append() calls
I20260620 02:00:19.640512 28394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:00:19.642921 28394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10: Bootstrap complete.
I20260620 02:00:19.650892 28394 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } }
I20260620 02:00:19.651965 28394 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4c0714eed64d479e0e70654e1e6a10, State: Initialized, Role: FOLLOWER
I20260620 02:00:19.653374 28394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } }
I20260620 02:00:19.654469 28400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:19.654850 28400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:19.655916 28394 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:00:19.681607 28411 catalog_manager.cc:1294] Loaded cluster ID: a06cac6b748c44a980d6931aeb6f244c
I20260620 02:00:19.681897 28411 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10: loading cluster ID for follower catalog manager: success
I20260620 02:00:19.689867 28411 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10: acquiring CA information for follower catalog manager: success
I20260620 02:00:19.699214 28411 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Jun 20 02:00:19 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920819, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:00:19.800231 28314 heartbeater.cc:499] Master 127.20.1.190:36605 was elected leader, sending a full tablet report...
I20260620 02:00:19.803840 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 28176
I20260620 02:00:19.849423 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:43833
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935
--webserver_port=33517
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935,127.20.1.187:37251 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:00:19.856745 28414 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:19.801742 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:50695 (local address 127.20.1.190:36605)
0620 02:00:19.802000 (+ 258us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:19.802008 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:19.847832 (+ 45824us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:50695: BlockingRecv error: recv got EOF from 127.20.1.129:50695 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 02:00:19.872342 28326 master_runner.cc:264] No leader master found from master 2e4c0714eed64d479e0e70654e1e6a10
I20260620 02:00:19.872565 28326 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:00:19.956789 28420 raft_consensus.cc:493] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2e4c0714eed64d479e0e70654e1e6a10)
I20260620 02:00:19.957201 28420 raft_consensus.cc:515] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } }
I20260620 02:00:19.960142 28420 leader_election.cc:290] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e4c0714eed64d479e0e70654e1e6a10 (127.20.1.190:36605), ade4889be736487b89bccec2e6e336ad (127.20.1.189:44283), 040c6f27fd3f4093b2d18cd433480f6c (127.20.1.187:37251)
I20260620 02:00:19.970156 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aad24f3b73c441988ddb1b60cf335584" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ade4889be736487b89bccec2e6e336ad" is_pre_election: true
I20260620 02:00:19.981073 28133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aad24f3b73c441988ddb1b60cf335584" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "040c6f27fd3f4093b2d18cd433480f6c" is_pre_election: true
I20260620 02:00:19.982384 27719 leader_election.cc:304] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: aad24f3b73c441988ddb1b60cf335584; no voters: 040c6f27fd3f4093b2d18cd433480f6c, ade4889be736487b89bccec2e6e336ad
I20260620 02:00:19.984328 28420 raft_consensus.cc:2749] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:00:19.986685 28369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aad24f3b73c441988ddb1b60cf335584" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2e4c0714eed64d479e0e70654e1e6a10" is_pre_election: true
I20260620 02:00:19.987617 28369 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad24f3b73c441988ddb1b60cf335584 in term 1.
I20260620 02:00:20.018663 28423 raft_consensus.cc:493] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2e4c0714eed64d479e0e70654e1e6a10)
I20260620 02:00:20.019053 28423 raft_consensus.cc:515] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } }
I20260620 02:00:20.030964 28423 leader_election.cc:290] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e4c0714eed64d479e0e70654e1e6a10 (127.20.1.190:36605), ade4889be736487b89bccec2e6e336ad (127.20.1.189:44283), aad24f3b73c441988ddb1b60cf335584 (127.20.1.188:36935)
I20260620 02:00:20.055464 28369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "040c6f27fd3f4093b2d18cd433480f6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2e4c0714eed64d479e0e70654e1e6a10" is_pre_election: true
I20260620 02:00:20.055867 28369 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 040c6f27fd3f4093b2d18cd433480f6c in term 1.
I20260620 02:00:20.060793 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "040c6f27fd3f4093b2d18cd433480f6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ade4889be736487b89bccec2e6e336ad" is_pre_election: true
I20260620 02:00:20.063663 27743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "040c6f27fd3f4093b2d18cd433480f6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "aad24f3b73c441988ddb1b60cf335584" is_pre_election: true
I20260620 02:00:20.064014 27743 raft_consensus.cc:2468] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 040c6f27fd3f4093b2d18cd433480f6c in term 1.
I20260620 02:00:20.064715 28110 leader_election.cc:304] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 040c6f27fd3f4093b2d18cd433480f6c, 2e4c0714eed64d479e0e70654e1e6a10, aad24f3b73c441988ddb1b60cf335584; no voters: ade4889be736487b89bccec2e6e336ad
I20260620 02:00:20.065971 28423 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 02:00:20.066529 28423 raft_consensus.cc:493] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2e4c0714eed64d479e0e70654e1e6a10)
I20260620 02:00:20.066679 28423 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:00:20.070652 28423 raft_consensus.cc:515] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } }
I20260620 02:00:20.072508 27743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "040c6f27fd3f4093b2d18cd433480f6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "aad24f3b73c441988ddb1b60cf335584"
I20260620 02:00:20.072767 27743 raft_consensus.cc:3060] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:00:20.072742 28369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "040c6f27fd3f4093b2d18cd433480f6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2e4c0714eed64d479e0e70654e1e6a10"
I20260620 02:00:20.072984 28369 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:00:20.076243 27743 raft_consensus.cc:2468] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 040c6f27fd3f4093b2d18cd433480f6c in term 2.
I20260620 02:00:20.077842 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "040c6f27fd3f4093b2d18cd433480f6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ade4889be736487b89bccec2e6e336ad"
I20260620 02:00:20.078866 28423 leader_election.cc:290] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [CANDIDATE]: Term 2 election: Requested vote from peers 2e4c0714eed64d479e0e70654e1e6a10 (127.20.1.190:36605), ade4889be736487b89bccec2e6e336ad (127.20.1.189:44283), aad24f3b73c441988ddb1b60cf335584 (127.20.1.188:36935)
I20260620 02:00:20.082080 28369 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 040c6f27fd3f4093b2d18cd433480f6c in term 2.
I20260620 02:00:20.082914 28109 leader_election.cc:304] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 040c6f27fd3f4093b2d18cd433480f6c, 2e4c0714eed64d479e0e70654e1e6a10, aad24f3b73c441988ddb1b60cf335584; no voters: ade4889be736487b89bccec2e6e336ad
I20260620 02:00:20.083411 28423 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:00:20.084442 28423 raft_consensus.cc:697] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [term 2 LEADER]: Becoming Leader. State: Replica: 040c6f27fd3f4093b2d18cd433480f6c, State: Running, Role: LEADER
I20260620 02:00:20.085130 28423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } }
I20260620 02:00:20.090088 28427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 040c6f27fd3f4093b2d18cd433480f6c. Latest consensus state: current_term: 2 leader_uuid: "040c6f27fd3f4093b2d18cd433480f6c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:20.090608 28427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:20.096186 28429 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:00:20.099606 28429 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:00:20.102650 28429 catalog_manager.cc:1294] Loaded cluster ID: a06cac6b748c44a980d6931aeb6f244c
I20260620 02:00:20.103426 28429 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:00:20.106963 28429 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:00:20.108532 28429 catalog_manager.cc:6080] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c: Loaded TSK: 0
I20260620 02:00:20.109838 28429 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 02:00:20.331845 28430 raft_consensus.cc:493] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2e4c0714eed64d479e0e70654e1e6a10)
I20260620 02:00:20.332401 28430 raft_consensus.cc:515] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } }
I20260620 02:00:20.335632 27743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ade4889be736487b89bccec2e6e336ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "aad24f3b73c441988ddb1b60cf335584" is_pre_election: true
I20260620 02:00:20.336158 27743 raft_consensus.cc:2393] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ade4889be736487b89bccec2e6e336ad in current term 2: Already voted for candidate 040c6f27fd3f4093b2d18cd433480f6c in this term.
I20260620 02:00:20.338408 28430 leader_election.cc:290] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e4c0714eed64d479e0e70654e1e6a10 (127.20.1.190:36605), aad24f3b73c441988ddb1b60cf335584 (127.20.1.188:36935), 040c6f27fd3f4093b2d18cd433480f6c (127.20.1.187:37251)
I20260620 02:00:20.349747 28369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ade4889be736487b89bccec2e6e336ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2e4c0714eed64d479e0e70654e1e6a10" is_pre_election: true
I20260620 02:00:20.350215 28369 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ade4889be736487b89bccec2e6e336ad in current term 2: Already voted for candidate 040c6f27fd3f4093b2d18cd433480f6c in this term.
I20260620 02:00:20.350474 28133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ade4889be736487b89bccec2e6e336ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "040c6f27fd3f4093b2d18cd433480f6c" is_pre_election: true
I20260620 02:00:20.351323 27641 leader_election.cc:304] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 1 yes votes; 3 no votes. yes voters: ade4889be736487b89bccec2e6e336ad; no voters: 040c6f27fd3f4093b2d18cd433480f6c, 2e4c0714eed64d479e0e70654e1e6a10, aad24f3b73c441988ddb1b60cf335584
I20260620 02:00:20.351835 28430 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:00:20.355559 28430 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 02:00:20.418912 28419 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:20.419641 28419 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:20.420025 28419 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:20.434062 28419 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:20.434295 28419 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:00:20.449820 28419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:43833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=33517
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935,127.20.1.187:37251
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:20.453601 28419 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:20.455495 28419 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:20.470999 28440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:20.470920 28438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:20.470970 28437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:20.471796 28419 server_base.cc:1061] running on GCE node
I20260620 02:00:20.472658 28419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:20.474067 28419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:20.475365 28419 hybrid_clock.cc:648] HybridClock initialized: now 1781920820475298 us; error 58 us; skew 500 ppm
I20260620 02:00:20.475848 28419 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:20 dist-test-slave-wg14 krb5kdc[27531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920820, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:20.481849 28419 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:00:20.485510 28419 webserver.cc:533] Webserver started at http://127.20.1.129:33517/ using document root <none> and password file <none>
I20260620 02:00:20.486594 28419 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:20.486748 28419 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:20.494647 28419 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:00:20.497056 27743 raft_consensus.cc:1275] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [term 2 FOLLOWER]: Refusing update from remote peer 040c6f27fd3f4093b2d18cd433480f6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260620 02:00:20.498097 28427 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 02:00:20.498874 28447 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:20.502446 28448 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 040c6f27fd3f4093b2d18cd433480f6c. Latest consensus state: current_term: 2 leader_uuid: "040c6f27fd3f4093b2d18cd433480f6c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:20.502671 28419 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s
I20260620 02:00:20.502800 28448 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:20.502925 28419 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "e27cbc9392f244b0a428f83525772354"
format_stamp: "Formatted at 2026-06-20 02:00:15 on dist-test-slave-wg14"
I20260620 02:00:20.504068 28419 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:20.544963 28419 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:20.550657 28419 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:20.550972 28419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:20.553012 28419 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:00:20.555655 28419 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:00:20.555804 28419 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:20.555903 28419 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:00:20.555974 28419 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:20.613296 28419 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:43833
I20260620 02:00:20.613356 28561 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:43833 every 8 connection(s)
I20260620 02:00:20.615418 28419 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:00:20.618306 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 28419
Jun 20 02:00:20 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920820, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
Jun 20 02:00:20 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920820, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:00:20.661418 28562 heartbeater.cc:344] Connected to a master server at 127.20.1.187:37251
I20260620 02:00:20.662703 28562 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:20.664057 28562 heartbeater.cc:507] Master 127.20.1.187:37251 requested a full tablet report, sending...
I20260620 02:00:20.667037 28123 ts_manager.cc:194] Re-registered known tserver with Master: e27cbc9392f244b0a428f83525772354 (127.20.1.129:43833)
I20260620 02:00:20.668712 28564 heartbeater.cc:344] Connected to a master server at 127.20.1.190:36605
I20260620 02:00:20.668954 28564 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:20.669477 28123 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:40119
I20260620 02:00:20.669584 28564 heartbeater.cc:507] Master 127.20.1.190:36605 requested a full tablet report, sending...
I20260620 02:00:20.672281 28359 ts_manager.cc:194] Registered new tserver with Master: e27cbc9392f244b0a428f83525772354 (127.20.1.129:43833)
I20260620 02:00:20.675352 28369 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [term 2 FOLLOWER]: Refusing update from remote peer 040c6f27fd3f4093b2d18cd433480f6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260620 02:00:20.677290 28427 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 02:00:20.687690 28574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 040c6f27fd3f4093b2d18cd433480f6c. Latest consensus state: current_term: 2 leader_uuid: "040c6f27fd3f4093b2d18cd433480f6c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:20.688109 28574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 02:00:20 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920820, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 02:00:20 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920820, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:00:20.695077 27664 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [term 2 FOLLOWER]: Refusing update from remote peer 040c6f27fd3f4093b2d18cd433480f6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260620 02:00:20.707541 28580 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260620 02:00:20.714941 28448 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "040c6f27fd3f4093b2d18cd433480f6c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:20.715306 28448 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad24f3b73c441988ddb1b60cf335584 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:20.717917 28563 heartbeater.cc:344] Connected to a master server at 127.20.1.188:36935
I20260620 02:00:20.718180 28563 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:20.718796 28563 heartbeater.cc:507] Master 127.20.1.188:36935 requested a full tablet report, sending...
I20260620 02:00:20.719955 28577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "040c6f27fd3f4093b2d18cd433480f6c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:20.719866 28578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "040c6f27fd3f4093b2d18cd433480f6c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:20.720263 28577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:20.720381 28578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:20.721174 27734 ts_manager.cc:194] Re-registered known tserver with Master: e27cbc9392f244b0a428f83525772354 (127.20.1.129:43833)
I20260620 02:00:20.726514 28565 heartbeater.cc:344] Connected to a master server at 127.20.1.189:44283
I20260620 02:00:20.726758 28565 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:20.727195 28565 heartbeater.cc:507] Master 127.20.1.189:44283 requested a full tablet report, sending...
I20260620 02:00:20.729413 28574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "040c6f27fd3f4093b2d18cd433480f6c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:20.729735 28574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4c0714eed64d479e0e70654e1e6a10 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:20.730767 28587 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "040c6f27fd3f4093b2d18cd433480f6c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:20.731386 28587 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:20.731940 28430 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 040c6f27fd3f4093b2d18cd433480f6c. Latest consensus state: current_term: 2 leader_uuid: "040c6f27fd3f4093b2d18cd433480f6c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:20.732152 28430 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade4889be736487b89bccec2e6e336ad [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:20.731408 27654 ts_manager.cc:194] Re-registered known tserver with Master: e27cbc9392f244b0a428f83525772354 (127.20.1.129:43833)
W20260620 02:00:20.738238 20486 mini_kdc.cc:336] Time spent kdestroy: real 0.120s user 0.005s sys 0.022s
I20260620 02:00:20.743575 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 28419
W20260620 02:00:20.746833 28589 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:20.740375 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:41808 (local address 127.20.1.129:43833)
0620 02:00:20.741323 (+ 948us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:20.741336 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:20.741378 (+ 42us) server_negotiation.cc:408] Connection header received
0620 02:00:20.741478 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:00:20.741486 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:00:20.741755 (+ 269us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:00:20.741963 (+ 208us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:00:20.743328 (+ 1365us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41808: BlockingRecv error: recv got EOF from 127.0.0.1:41808 (error 108)
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":566,"threads_started":1}
W20260620 02:00:20.748809 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 28419: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:43833: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:00:20.749020 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 28419
I20260620 02:00:20.785940 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:43833
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935
--webserver_port=33517
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935,127.20.1.187:37251
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:00:21.267287 28581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "040c6f27fd3f4093b2d18cd433480f6c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2e4c0714eed64d479e0e70654e1e6a10" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 36605 } } peers { permanent_uuid: "ade4889be736487b89bccec2e6e336ad" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 44283 } } peers { permanent_uuid: "aad24f3b73c441988ddb1b60cf335584" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 36935 } } peers { permanent_uuid: "040c6f27fd3f4093b2d18cd433480f6c" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37251 } attrs { promote: false } } }
I20260620 02:00:21.267761 28581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c [sys.catalog]: This master's current role is: LEADER
W20260620 02:00:21.290341 28591 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:21.290748 28591 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:21.290861 28591 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 02:00:21.290912 28591 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:21.302691 28591 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:21.302989 28591 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:00:21.315930 28591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:43833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=33517
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935,127.20.1.187:37251
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:21.319582 28591 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:21.322309 28591 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:21.339180 28599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:21.339318 28602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:21.339326 28600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:21.342231 28591 server_base.cc:1061] running on GCE node
I20260620 02:00:21.343434 28591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:21.346248 28591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:21.349931 28591 hybrid_clock.cc:648] HybridClock initialized: now 1781920821349812 us; error 328 us; skew 500 ppm
I20260620 02:00:21.350389 28591 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:21 dist-test-slave-wg14 krb5kdc[27531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920821, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:21.354826 28591 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:00:21.357836 28591 webserver.cc:533] Webserver started at http://127.20.1.129:33517/ using document root <none> and password file <none>
I20260620 02:00:21.358906 28591 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:21.359047 28591 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:21.368880 28591 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 02:00:21.374176 28609 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:21.378391 28591 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260620 02:00:21.378953 28591 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "e27cbc9392f244b0a428f83525772354"
format_stamp: "Formatted at 2026-06-20 02:00:15 on dist-test-slave-wg14"
I20260620 02:00:21.380861 28591 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:21.425985 28591 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:21.432207 28591 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:21.432525 28591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:21.434201 28591 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:00:21.436709 28591 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:00:21.436828 28591 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:21.436935 28591 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:00:21.436980 28591 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 02:00:21.437438 28591 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 02:00:21.437529 28591 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 02:00:21.437608 28591 fault_injection.cc:59] injecting fault for kudu (pid 28591): FLAGS_fault_before_start (process will exit)
I20260620 02:00:21.459803 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:43833
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935
--webserver_port=33517
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935,127.20.1.187:37251 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:00:21.962821 28617 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:21.963601 28617 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:21.963789 28617 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:21.976668 28617 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:21.977392 28617 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:00:21.996863 28617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:43833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=33517
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:36605,127.20.1.189:44283,127.20.1.188:36935,127.20.1.187:37251
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:21.999876 28617 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:22.001935 28617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:22.019472 28623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:22.019870 28622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:22.021502 28625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:22.022454 28617 server_base.cc:1061] running on GCE node
I20260620 02:00:22.023722 28617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:22.025280 28617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:22.026614 28617 hybrid_clock.cc:648] HybridClock initialized: now 1781920822026551 us; error 58 us; skew 500 ppm
I20260620 02:00:22.027029 28617 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:22 dist-test-slave-wg14 krb5kdc[27531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920822, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:22.032141 28617 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:00:22.037897 28617 webserver.cc:533] Webserver started at http://127.20.1.129:33517/ using document root <none> and password file <none>
I20260620 02:00:22.038906 28617 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:22.039077 28617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:22.054724 28617 fs_manager.cc:714] Time spent opening directory manager: real 0.012s user 0.010s sys 0.001s
I20260620 02:00:22.059366 28632 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:22.062583 28617 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260620 02:00:22.062849 28617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "e27cbc9392f244b0a428f83525772354"
format_stamp: "Formatted at 2026-06-20 02:00:15 on dist-test-slave-wg14"
I20260620 02:00:22.064060 28617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:22.088302 28617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:22.098505 28617 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:22.098898 28617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:22.100229 28617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:00:22.102628 28617 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:00:22.102828 28617 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:22.102999 28617 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:00:22.103080 28617 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:22.181074 28617 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:43833
I20260620 02:00:22.181139 28744 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:43833 every 8 connection(s)
I20260620 02:00:22.183728 28617 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:00:22.192476 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 28617
I20260620 02:00:22.192862 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 28617
Jun 20 02:00:22 dist-test-slave-wg14 krb5kdc[27531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920822, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:00:22.228920 28746 heartbeater.cc:344] Connected to a master server at 127.20.1.188:36935
I20260620 02:00:22.229367 28746 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:22.231503 28746 heartbeater.cc:507] Master 127.20.1.188:36935 requested a full tablet report, sending...
W20260620 02:00:22.235523 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 28617: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:43833: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:00:22.235877 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 28617
I20260620 02:00:22.237758 27734 ts_manager.cc:194] Re-registered known tserver with Master: e27cbc9392f244b0a428f83525772354 (127.20.1.129:43833)
W20260620 02:00:22.259294 27721 connection.cc:573] server connection from 127.20.1.129:44297 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 02:00:22.259677 28751 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:22.191671 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:36977 (local address 127.20.1.190:36605)
0620 02:00:22.221917 (+ 30246us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:22.221925 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:22.222823 (+ 898us) server_negotiation.cc:408] Connection header received
0620 02:00:22.222914 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:00:22.222921 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:00:22.223032 (+ 111us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:00:22.223649 (+ 617us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:00:22.231635 (+ 7986us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 02:00:22.237798 (+ 6163us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:00:22.259290 (+ 21492us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:36977: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":30064,"thread_start_us":31003,"threads_started":1}
W20260620 02:00:22.260435 28758 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:22.224985 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:43735 (local address 127.20.1.189:44283)
0620 02:00:22.226325 (+ 1340us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:22.226332 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:22.227448 (+ 1116us) server_negotiation.cc:408] Connection header received
0620 02:00:22.227585 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:00:22.227592 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:00:22.227963 (+ 371us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:00:22.237798 (+ 9835us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:00:22.259289 (+ 21491us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:43735: BlockingRecv error: recv got EOF from 127.20.1.129:43735 (error 108)
Metrics: {"server-negotiator.queue_time_us":1195,"thread_start_us":1573,"threads_started":1}
W20260620 02:00:22.260464 28750 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:22.191293 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:46501 (local address 127.20.1.187:37251)
0620 02:00:22.225420 (+ 34127us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:22.225427 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:22.225449 (+ 22us) server_negotiation.cc:408] Connection header received
0620 02:00:22.225541 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:00:22.225547 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:00:22.225633 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:00:22.225765 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:00:22.228358 (+ 2593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 02:00:22.231196 (+ 2838us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:00:22.259282 (+ 28086us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:46501: BlockingRecv error: recv got EOF from 127.20.1.129:46501 (error 108)
Metrics: {"server-negotiator.queue_time_us":33992,"thread_start_us":34252,"threads_started":1}
I20260620 02:00:22.260624 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 28326
W20260620 02:00:22.263520 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 28326: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:36605: server requires authentication, but client does not have Kerberos credentials available
W20260620 02:00:22.263644 28751 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:22.261733 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:42914 (local address 127.20.1.190:36605)
0620 02:00:22.261964 (+ 231us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:22.261969 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:22.261989 (+ 20us) server_negotiation.cc:408] Connection header received
0620 02:00:22.262151 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:00:22.262157 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:00:22.262237 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:00:22.262348 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:00:22.263402 (+ 1054us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42914: BlockingRecv error: recv got EOF from 127.0.0.1:42914 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 02:00:22.263907 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 28326
I20260620 02:00:22.292136 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 27623
I20260620 02:00:22.294859 27685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:00:22.438706 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 27623
I20260620 02:00:22.464834 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 27700
I20260620 02:00:22.466298 27764 generic_service.cc:196] Checking for leaks (request via RPC)
W20260620 02:00:22.624418 28109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.190:36605: connect: Connection refused (error 111)
W20260620 02:00:22.626753 28109 consensus_peers.cc:597] T 00000000000000000000000000000000 P 040c6f27fd3f4093b2d18cd433480f6c -> Peer 2e4c0714eed64d479e0e70654e1e6a10 (127.20.1.190:36605): Couldn't send request to peer 2e4c0714eed64d479e0e70654e1e6a10. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:36605: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 02:00:22.658385 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 27700
I20260620 02:00:22.682076 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 27981
I20260620 02:00:22.684188 28153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:00:22.843772 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 27981
2026-06-20T02:00:22Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/7 (15255 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/8
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:00:23 dist-test-slave-wg14 krb5kdc[28767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:00:23 dist-test-slave-wg14 krb5kdc[28767](info): set up 2 sockets
Jun 20 02:00:23 dist-test-slave-wg14 krb5kdc[28767](info): commencing operation
krb5kdc: starting...
W20260620 02:00:25.627871 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.729s user 0.010s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 02:00:25 dist-test-slave-wg14 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:00:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:00:25Z Disabled control of system clock
2026-06-20T02:00:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:00:25Z Disabled control of system clock
2026-06-20T02:00:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:00:26Z Disabled control of system clock
2026-06-20T02:00:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:00:26Z Disabled control of system clock
2026-06-20T02:00:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:00:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:00:26.282701 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.190: real 0.104s user 0.000s sys 0.032s
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:00:26.388595 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.190: real 0.106s user 0.000s sys 0.028s
I20260620 02:00:26.390599 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:33851
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:00:27.017876 28791 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:27.018287 28791 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:27.018440 28791 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:27.035302 28791 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:00:27.035516 28791 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:27.035722 28791 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:00:27.035997 28791 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:00:27.052711 28791 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:33851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:27.057350 28791 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:27.059394 28791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:27.072376 28796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:27.072708 28797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:27.073907 28799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:27.076514 28791 server_base.cc:1061] running on GCE node
I20260620 02:00:27.077761 28791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:27.080477 28791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:27.084195 28791 hybrid_clock.cc:648] HybridClock initialized: now 1781920827083926 us; error 162 us; skew 500 ppm
I20260620 02:00:27.084769 28791 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:27 dist-test-slave-wg14 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920827, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:27.088814 28791 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:00:27.096626 28791 webserver.cc:533] Webserver started at http://127.20.1.190:46711/ using document root <none> and password file <none>
I20260620 02:00:27.097827 28791 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:27.098006 28791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:27.098444 28791 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:00:27.101471 28791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "c110337d7e3b4d4db476c1ad3fd5e40e"
format_stamp: "Formatted at 2026-06-20 02:00:27 on dist-test-slave-wg14"
I20260620 02:00:27.102500 28791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "c110337d7e3b4d4db476c1ad3fd5e40e"
format_stamp: "Formatted at 2026-06-20 02:00:27 on dist-test-slave-wg14"
I20260620 02:00:27.114079 28791 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.011s sys 0.000s
I20260620 02:00:27.121486 28806 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:27.124127 28791 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20260620 02:00:27.124399 28791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "c110337d7e3b4d4db476c1ad3fd5e40e"
format_stamp: "Formatted at 2026-06-20 02:00:27 on dist-test-slave-wg14"
I20260620 02:00:27.124693 28791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:27.150550 28791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:27.160790 28791 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:27.161145 28791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:27.195349 28791 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:33851
I20260620 02:00:27.195349 28857 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:33851 every 8 connection(s)
I20260620 02:00:27.199214 28791 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:00:27.200819 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 28791
I20260620 02:00:27.201097 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:00:27.211517 28858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:00:27.225263 28858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.189@KRBTEST.COM" created.
Jun 20 02:00:27 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920827, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:00:27.277882 28858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } has no permanent_uuid. Determining permanent_uuid...
W20260620 02:00:27.279707 28807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:34413: connect: Connection refused (error 111)
W20260620 02:00:27.281854 28858 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:34413: Network error: Client connection negotiation failed: client connection to 127.20.1.189:34413: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:00:27.313447 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.189: real 0.111s user 0.001s sys 0.032s
I20260620 02:00:27.331441 28858 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } attempt: 1
W20260620 02:00:27.334889 28858 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:34413: Network error: Client connection negotiation failed: client connection to 127.20.1.189:34413: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:00:27.397526 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:34413
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:00:27.403287 28858 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } attempt: 2
W20260620 02:00:27.406031 28858 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:34413: Network error: Client connection negotiation failed: client connection to 127.20.1.189:34413: connect: Connection refused (error 111)
I20260620 02:00:27.497506 28858 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } attempt: 3
W20260620 02:00:27.500885 28858 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:34413: Network error: Client connection negotiation failed: client connection to 127.20.1.189:34413: connect: Connection refused (error 111)
I20260620 02:00:27.644317 28858 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } attempt: 4
W20260620 02:00:27.648715 28858 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:34413: Network error: Client connection negotiation failed: client connection to 127.20.1.189:34413: connect: Connection refused (error 111)
W20260620 02:00:27.871774 28867 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:27.872191 28867 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:27.872291 28867 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:27.882704 28867 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:00:27.882861 28867 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:27.882922 28867 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:00:27.882974 28867 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:00:27.897241 28867 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:34413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:27.899338 28867 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:27.903440 28867 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:27.920792 28873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:27.920820 28872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:27.921247 28875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:27.923425 28867 server_base.cc:1061] running on GCE node
I20260620 02:00:27.925128 28867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:27.928308 28867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:27.931866 28867 hybrid_clock.cc:648] HybridClock initialized: now 1781920827931411 us; error 355 us; skew 500 ppm
I20260620 02:00:27.932456 28867 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:27 dist-test-slave-wg14 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920827, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:27.937674 28867 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 02:00:27.941514 28867 webserver.cc:533] Webserver started at http://127.20.1.189:38585/ using document root <none> and password file <none>
I20260620 02:00:27.942870 28867 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:27.943053 28867 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:27.943879 28867 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:00:27.947029 28867 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "429d9d90eb8f4700a2adcd38da86dc6e"
format_stamp: "Formatted at 2026-06-20 02:00:27 on dist-test-slave-wg14"
I20260620 02:00:27.948017 28858 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } attempt: 5
I20260620 02:00:27.948530 28867 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "429d9d90eb8f4700a2adcd38da86dc6e"
format_stamp: "Formatted at 2026-06-20 02:00:27 on dist-test-slave-wg14"
W20260620 02:00:27.951659 28858 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:34413: Network error: Client connection negotiation failed: client connection to 127.20.1.189:34413: connect: Connection refused (error 111)
I20260620 02:00:27.956763 28867 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260620 02:00:27.961846 28882 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:27.964604 28867 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 02:00:27.964902 28867 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "429d9d90eb8f4700a2adcd38da86dc6e"
format_stamp: "Formatted at 2026-06-20 02:00:27 on dist-test-slave-wg14"
I20260620 02:00:27.965188 28867 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:27.994378 28867 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:28.000901 28867 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:28.001276 28867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:28.033275 28867 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:34413
I20260620 02:00:28.033694 28933 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:34413 every 8 connection(s)
I20260620 02:00:28.035648 28867 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 02:00:28.039386 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 28867
I20260620 02:00:28.039693 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 02:00:28.042680 28934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:00:28.052321 28934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:28 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920827, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.188@KRBTEST.COM" created.
I20260620 02:00:28.088337 28934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:28 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920827, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:00:28.122714 28934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } has no permanent_uuid. Determining permanent_uuid...
W20260620 02:00:28.124850 28885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34651: connect: Connection refused (error 111)
W20260620 02:00:28.130240 28934 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34651: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34651: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:00:28.141048 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.188: real 0.100s user 0.004s sys 0.032s
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.188@KRBTEST.COM" created.
I20260620 02:00:28.179700 28934 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } attempt: 1
W20260620 02:00:28.184019 28934 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34651: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34651: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:00:28.227756 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:34651
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:00:28.252416 28934 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } attempt: 2
W20260620 02:00:28.256755 28934 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34651: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34651: connect: Connection refused (error 111)
I20260620 02:00:28.348089 28934 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } attempt: 3
W20260620 02:00:28.351707 28934 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34651: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34651: connect: Connection refused (error 111)
I20260620 02:00:28.495087 28934 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } attempt: 4
I20260620 02:00:28.498984 28858 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } attempt: 6
W20260620 02:00:28.503228 28934 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34651: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34651: connect: Connection refused (error 111)
Jun 20 02:00:28 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920827, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:00:28.527931 28858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } has no permanent_uuid. Determining permanent_uuid...
W20260620 02:00:28.530530 28858 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34651: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34651: connect: Connection refused (error 111)
I20260620 02:00:28.582854 28858 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } attempt: 1
W20260620 02:00:28.585964 28858 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34651: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34651: connect: Connection refused (error 111)
I20260620 02:00:28.660356 28858 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } attempt: 2
W20260620 02:00:28.663793 28858 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34651: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34651: connect: Connection refused (error 111)
W20260620 02:00:28.756692 28944 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:28.757546 28944 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:28.757694 28944 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:28.769512 28944 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:00:28.769666 28944 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:28.769724 28944 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:00:28.769791 28944 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:00:28.777546 28858 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } attempt: 3
W20260620 02:00:28.781168 28858 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34651: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34651: connect: Connection refused (error 111)
I20260620 02:00:28.784689 28944 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:34651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:28.786757 28944 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:28.789018 28944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:28.801465 28951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:28.801494 28952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:28.802556 28934 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } attempt: 5
W20260620 02:00:28.803382 28954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:28.804507 28944 server_base.cc:1061] running on GCE node
W20260620 02:00:28.805368 28934 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:34651: Network error: Client connection negotiation failed: client connection to 127.20.1.188:34651: connect: Connection refused (error 111)
I20260620 02:00:28.806041 28944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:28.808564 28944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:28.812247 28944 hybrid_clock.cc:648] HybridClock initialized: now 1781920828811996 us; error 151 us; skew 500 ppm
I20260620 02:00:28.812814 28944 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:28 dist-test-slave-wg14 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920828, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:28.818197 28944 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 02:00:28.821484 28944 webserver.cc:533] Webserver started at http://127.20.1.188:40405/ using document root <none> and password file <none>
I20260620 02:00:28.823195 28944 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:28.823385 28944 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:28.823841 28944 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:00:28.828492 28944 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "9aa09029e9694539a419a147fd3a9e15"
format_stamp: "Formatted at 2026-06-20 02:00:28 on dist-test-slave-wg14"
I20260620 02:00:28.829658 28944 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "9aa09029e9694539a419a147fd3a9e15"
format_stamp: "Formatted at 2026-06-20 02:00:28 on dist-test-slave-wg14"
I20260620 02:00:28.837591 28944 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260620 02:00:28.843099 28961 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:28.845449 28944 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:00:28.845660 28944 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "9aa09029e9694539a419a147fd3a9e15"
format_stamp: "Formatted at 2026-06-20 02:00:28 on dist-test-slave-wg14"
I20260620 02:00:28.845928 28944 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:28.877822 28944 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:28.886855 28944 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:28.887323 28944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:28.915119 28944 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:34651
I20260620 02:00:28.915154 29012 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:34651 every 8 connection(s)
I20260620 02:00:28.917024 28944 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 02:00:28.918435 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 28944
I20260620 02:00:28.918753 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 02:00:28.925006 29013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:00:28.930513 28858 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } attempt: 4
Jun 20 02:00:28 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:00:28.934435 29013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:28 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920827, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 02:00:28 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 02:00:28 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920828, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:00:28.977242 28858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e: Bootstrap starting.
Jun 20 02:00:28 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:00:28.986781 29013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:00:28.988054 28858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e: Neither blocks nor log segments found. Creating new log.
I20260620 02:00:28.990042 28858 log.cc:826] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:00:28.994082 28858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e: No bootstrap required, opened a new log
Jun 20 02:00:28 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920828, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:00:29.009123 28858 raft_consensus.cc:359] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } }
I20260620 02:00:29.009629 28858 raft_consensus.cc:385] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:00:29.009831 28858 raft_consensus.cc:740] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c110337d7e3b4d4db476c1ad3fd5e40e, State: Initialized, Role: FOLLOWER
I20260620 02:00:29.010752 28858 consensus_queue.cc:260] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } }
I20260620 02:00:29.012063 29013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:00:29.014789 29022 sys_catalog.cc:455] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } }
I20260620 02:00:29.015726 29022 sys_catalog.cc:458] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:29.018793 28858 sys_catalog.cc:565] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 02:00:29 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920828, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:00:29.033753 29022 raft_consensus.cc:493] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:00:29.034055 29022 raft_consensus.cc:515] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } }
I20260620 02:00:29.036198 29022 leader_election.cc:290] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429d9d90eb8f4700a2adcd38da86dc6e (127.20.1.189:34413), 9aa09029e9694539a419a147fd3a9e15 (127.20.1.188:34651)
I20260620 02:00:29.037751 28791 master_runner.cc:187] Error getting master registration for 127.20.1.189:34413: OK, instance_id { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" instance_seqno: 1781920828025209 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:00:29.049495 28791 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:00:29.054973 29013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15: Bootstrap starting.
I20260620 02:00:29.066164 29013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15: Neither blocks nor log segments found. Creating new log.
I20260620 02:00:29.068353 29013 log.cc:826] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15: Log is configured to *not* fsync() on all Append() calls
W20260620 02:00:29.073812 29033 catalog_manager.cc:1593] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:00:29.074046 29033 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:00:29.083413 29013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15: No bootstrap required, opened a new log
I20260620 02:00:29.093900 29013 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } }
I20260620 02:00:29.094380 29013 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:00:29.094548 29013 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa09029e9694539a419a147fd3a9e15, State: Initialized, Role: FOLLOWER
I20260620 02:00:29.095686 29013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } }
I20260620 02:00:29.098208 29036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } }
I20260620 02:00:29.098845 29036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:29.099865 29013 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:00:29.099916 28988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa09029e9694539a419a147fd3a9e15" is_pre_election: true
I20260620 02:00:29.100806 28988 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c110337d7e3b4d4db476c1ad3fd5e40e in term 0.
I20260620 02:00:29.102079 28809 leader_election.cc:304] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9aa09029e9694539a419a147fd3a9e15, c110337d7e3b4d4db476c1ad3fd5e40e; no voters:
I20260620 02:00:29.102931 29022 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:00:29.103375 29022 raft_consensus.cc:493] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:00:29.103628 29022 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:00:29.107517 29022 raft_consensus.cc:515] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } }
I20260620 02:00:29.108651 29022 leader_election.cc:290] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [CANDIDATE]: Term 1 election: Requested vote from peers 429d9d90eb8f4700a2adcd38da86dc6e (127.20.1.189:34413), 9aa09029e9694539a419a147fd3a9e15 (127.20.1.188:34651)
I20260620 02:00:29.111788 28988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa09029e9694539a419a147fd3a9e15"
I20260620 02:00:29.112188 28988 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:00:29.116110 28988 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c110337d7e3b4d4db476c1ad3fd5e40e in term 1.
I20260620 02:00:29.117758 28809 leader_election.cc:304] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9aa09029e9694539a419a147fd3a9e15, c110337d7e3b4d4db476c1ad3fd5e40e; no voters:
I20260620 02:00:29.119959 29022 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:00:29.131680 29022 raft_consensus.cc:697] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 1 LEADER]: Becoming Leader. State: Replica: c110337d7e3b4d4db476c1ad3fd5e40e, State: Running, Role: LEADER
I20260620 02:00:29.132601 29022 consensus_queue.cc:237] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } }
I20260620 02:00:29.138732 29044 sys_catalog.cc:455] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c110337d7e3b4d4db476c1ad3fd5e40e. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } }
I20260620 02:00:29.139578 29044 sys_catalog.cc:458] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: This master's current role is: LEADER
W20260620 02:00:29.140239 28967 tablet.cc:2406] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:00:29.145044 29048 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:00:29.148584 29048 catalog_manager.cc:1519] Initializing Kudu cluster ID...
W20260620 02:00:29.173442 29050 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:00:29.173707 29050 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 02:00:29.174503 28812 tablet.cc:2406] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:00:29.185061 28988 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 1 FOLLOWER]: Refusing update from remote peer c110337d7e3b4d4db476c1ad3fd5e40e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:00:29.186929 29044 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:00:29.182276 28944 master_runner.cc:187] Error getting master registration for 127.20.1.189:34413: OK, instance_id { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" instance_seqno: 1781920828025209 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:00:29.240424 28944 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:00:29.254522 29036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c110337d7e3b4d4db476c1ad3fd5e40e. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } }
I20260620 02:00:29.255272 29036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:29.274133 29053 mvcc.cc:204] Tried to move back new op lower bound from 7298747716326289408 to 7298747716133961728. Current Snapshot: MvccSnapshot[applied={T|T < 7298747716326289408}]
I20260620 02:00:29.275756 29044 sys_catalog.cc:455] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } }
I20260620 02:00:29.276051 29044 sys_catalog.cc:458] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:29.281412 29036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } }
I20260620 02:00:29.282020 29036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:29.292229 29048 catalog_manager.cc:1382] Generated new cluster ID: 019cac4944f7460dbebf1976090a6846
I20260620 02:00:29.292407 29048 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:00:29.323323 29048 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:00:29.340063 29048 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:00:29.352862 28934 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } attempt: 6
Jun 20 02:00:29 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920827, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:00:29.407481 29048 catalog_manager.cc:6069] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e: Generated new TSK 0
I20260620 02:00:29.414572 29048 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 02:00:29.430558 28934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e: Bootstrap starting.
I20260620 02:00:29.444288 28934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e: Neither blocks nor log segments found. Creating new log.
I20260620 02:00:29.446447 28934 log.cc:826] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:00:29.466652 28934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e: No bootstrap required, opened a new log
I20260620 02:00:29.483253 28934 raft_consensus.cc:359] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } }
I20260620 02:00:29.483964 28934 raft_consensus.cc:385] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:00:29.484220 28934 raft_consensus.cc:740] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429d9d90eb8f4700a2adcd38da86dc6e, State: Initialized, Role: FOLLOWER
I20260620 02:00:29.485646 28934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } }
I20260620 02:00:29.495743 29060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } }
I20260620 02:00:29.502372 28934 sys_catalog.cc:565] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:00:29.502807 28907 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:00:29.504390 29060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: This master's current role is: FOLLOWER
W20260620 02:00:29.505860 28888 tablet.cc:2406] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:00:29.507324 28908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429d9d90eb8f4700a2adcd38da86dc6e"
I20260620 02:00:29.509415 28909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" is_pre_election: true
I20260620 02:00:29.506992 28907 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 1 FOLLOWER]: Refusing update from remote peer c110337d7e3b4d4db476c1ad3fd5e40e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:00:29.516043 29022 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [LEADER]: Connected to new peer: Peer: permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:00:29.565122 29066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } }
I20260620 02:00:29.565721 29066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:29.564219 29060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c110337d7e3b4d4db476c1ad3fd5e40e. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } }
I20260620 02:00:29.566397 29060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: This master's current role is: FOLLOWER
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
I20260620 02:00:29.616415 29076 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:00:29.670352 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.227s user 0.003s sys 0.042s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:00:29.766376 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:00:29.955788 29022 sys_catalog.cc:455] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } }
I20260620 02:00:29.956301 29022 sys_catalog.cc:458] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:30.176510 29050 catalog_manager.cc:1294] Loaded cluster ID: 019cac4944f7460dbebf1976090a6846
I20260620 02:00:30.176641 29050 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15: loading cluster ID for follower catalog manager: success
I20260620 02:00:30.179947 29050 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15: acquiring CA information for follower catalog manager: success
I20260620 02:00:30.182672 29050 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 02:00:30.359727 29080 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:30.360141 29080 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:30.360285 29080 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:30.376889 29080 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:30.377154 29080 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:00:30.397684 29080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:30.399823 29080 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:30.401965 29080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:30.417198 29087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:30.418292 29088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:30.421036 29090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:30.422013 29080 server_base.cc:1061] running on GCE node
I20260620 02:00:30.422957 29080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:30.424670 29080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:30.428350 29080 hybrid_clock.cc:648] HybridClock initialized: now 1781920830427994 us; error 257 us; skew 500 ppm
I20260620 02:00:30.428892 29080 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:30 dist-test-slave-wg14 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920830, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:30.434319 29080 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:00:30.439150 29080 webserver.cc:533] Webserver started at http://127.20.1.129:40607/ using document root <none> and password file <none>
I20260620 02:00:30.440116 29080 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:30.440274 29080 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:30.440698 29080 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:00:30.444566 29080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "127c5477390c4c31b6f6375e5b1c6e68"
format_stamp: "Formatted at 2026-06-20 02:00:30 on dist-test-slave-wg14"
I20260620 02:00:30.445827 29080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "127c5477390c4c31b6f6375e5b1c6e68"
format_stamp: "Formatted at 2026-06-20 02:00:30 on dist-test-slave-wg14"
I20260620 02:00:30.460223 29080 fs_manager.cc:696] Time spent creating directory manager: real 0.014s user 0.015s sys 0.000s
I20260620 02:00:30.465077 29097 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:30.468926 29080 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.004s sys 0.003s
I20260620 02:00:30.469172 29080 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "127c5477390c4c31b6f6375e5b1c6e68"
format_stamp: "Formatted at 2026-06-20 02:00:30 on dist-test-slave-wg14"
I20260620 02:00:30.469439 29080 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:30.505167 29080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:30.517071 29080 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:30.517449 29080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:30.518690 29080 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:00:30.520974 29080 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:00:30.521081 29080 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:30.521205 29080 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:00:30.521267 29080 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:30.598496 29080 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:45257
I20260620 02:00:30.598790 29209 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:45257 every 8 connection(s)
I20260620 02:00:30.600256 29080 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:00:30.602514 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 29080
I20260620 02:00:30.602774 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
I20260620 02:00:30.624960 29076 catalog_manager.cc:1294] Loaded cluster ID: 019cac4944f7460dbebf1976090a6846
I20260620 02:00:30.625124 29076 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e: loading cluster ID for follower catalog manager: success
I20260620 02:00:30.637588 29076 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e: acquiring CA information for follower catalog manager: success
I20260620 02:00:30.656399 29076 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Jun 20 02:00:30 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920830, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 02:00:30 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920830, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 02:00:30 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920830, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:00:30.676474 29211 heartbeater.cc:344] Connected to a master server at 127.20.1.188:34651
I20260620 02:00:30.676949 29211 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:30.678894 29211 heartbeater.cc:507] Master 127.20.1.188:34651 requested a full tablet report, sending...
I20260620 02:00:30.684579 28978 ts_manager.cc:194] Registered new tserver with Master: 127c5477390c4c31b6f6375e5b1c6e68 (127.20.1.129:45257)
I20260620 02:00:30.686542 29210 heartbeater.cc:344] Connected to a master server at 127.20.1.190:33851
I20260620 02:00:30.686718 29210 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:30.689057 29210 heartbeater.cc:507] Master 127.20.1.190:33851 requested a full tablet report, sending...
I20260620 02:00:30.697574 29212 heartbeater.cc:344] Connected to a master server at 127.20.1.189:34413
I20260620 02:00:30.697839 29212 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:30.698510 28823 ts_manager.cc:194] Registered new tserver with Master: 127c5477390c4c31b6f6375e5b1c6e68 (127.20.1.129:45257)
I20260620 02:00:30.698608 29212 heartbeater.cc:507] Master 127.20.1.189:34413 requested a full tablet report, sending...
I20260620 02:00:30.700811 28823 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:38175
I20260620 02:00:30.701004 28899 ts_manager.cc:194] Registered new tserver with Master: 127c5477390c4c31b6f6375e5b1c6e68 (127.20.1.129:45257)
I20260620 02:00:30.720985 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
WARNING: no policy specified for kudu/127.20.1.187@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.187@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.187 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.187 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.187@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.187@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.187 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.187 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:00:30.911697 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:37223
--webserver_interface=127.20.1.187
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.187
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651,127.20.1.187:37223 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:00:31.474707 29225 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:31.475085 29225 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:31.475257 29225 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:31.489434 29225 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:00:31.489575 29225 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:31.489799 29225 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:00:31.489910 29225 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:00:31.505935 29225 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651,127.20.1.187:37223
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.187
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:37223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.20.1.187
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:31.508490 29225 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:31.510733 29225 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:31.523967 29231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:31.524507 29230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:31.525672 29233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:31.527017 29225 server_base.cc:1061] running on GCE node
I20260620 02:00:31.528158 29225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:31.531342 29225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:31.532897 29225 hybrid_clock.cc:648] HybridClock initialized: now 1781920831532757 us; error 85 us; skew 500 ppm
I20260620 02:00:31.533421 29225 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:31 dist-test-slave-wg14 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920831, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:31.537663 29225 init.cc:377] Logged in from keytab as kudu/127.20.1.187@KRBTEST.COM (short username kudu)
I20260620 02:00:31.540241 29225 webserver.cc:533] Webserver started at http://127.20.1.187:38143/ using document root <none> and password file <none>
I20260620 02:00:31.541884 29225 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:31.542222 29225 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:31.542866 29225 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:00:31.548271 29225 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/data/instance:
uuid: "50669d11f58b4df087a03a1796667cb2"
format_stamp: "Formatted at 2026-06-20 02:00:31 on dist-test-slave-wg14"
I20260620 02:00:31.549360 29225 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/wal/instance:
uuid: "50669d11f58b4df087a03a1796667cb2"
format_stamp: "Formatted at 2026-06-20 02:00:31 on dist-test-slave-wg14"
I20260620 02:00:31.557507 29225 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260620 02:00:31.563948 29240 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:31.567261 29225 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.004s sys 0.002s
I20260620 02:00:31.567530 29225 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "50669d11f58b4df087a03a1796667cb2"
format_stamp: "Formatted at 2026-06-20 02:00:31 on dist-test-slave-wg14"
I20260620 02:00:31.568187 29225 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:31.588599 29225 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:31.593672 29225 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:31.594020 29225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:31.622829 29225 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:37223
I20260620 02:00:31.622792 29291 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:37223 every 8 connection(s)
I20260620 02:00:31.626327 29225 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 02:00:31.631887 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 29225
I20260620 02:00:31.632153 29292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:00:31.643743 29292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:31 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
Jun 20 02:00:31 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920831, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:00:31.674181 29292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:31 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920831, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:00:31.694185 29292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:31 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920831, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:00:31.706035 29210 heartbeater.cc:499] Master 127.20.1.190:33851 was elected leader, sending a full tablet report...
I20260620 02:00:31.715225 29292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:00:31 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920831, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
I20260620 02:00:31.746495 29292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: Bootstrap starting.
I20260620 02:00:31.750591 29292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: Neither blocks nor log segments found. Creating new log.
I20260620 02:00:31.751902 29292 log.cc:826] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:00:31.755753 29292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: No bootstrap required, opened a new log
I20260620 02:00:31.764645 29292 raft_consensus.cc:359] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } }
I20260620 02:00:31.765450 29292 raft_consensus.cc:385] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:00:31.765631 29292 raft_consensus.cc:740] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50669d11f58b4df087a03a1796667cb2, State: Initialized, Role: FOLLOWER
I20260620 02:00:31.766654 29292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } }
I20260620 02:00:31.767930 29301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } } }
I20260620 02:00:31.768275 29301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:31.769097 29292 sys_catalog.cc:565] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:00:31.781594 29312 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:00:31.781774 29312 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:00:31.793627 29225 master_runner.cc:429] Detected that this master 50669d11f58b4df087a03a1796667cb2 is joining an existing cluster
I20260620 02:00:31.793807 29225 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260620 02:00:31.794827 29225 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 02:00:31.795271 29225 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 02:00:31.795485 29225 tablet_replica.cc:333] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: stopping tablet replica
I20260620 02:00:31.798599 29225 master_runner.cc:434] Initiating AddMaster RPC to add 127.20.1.187:37223
Jun 20 02:00:31 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920827, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
I20260620 02:00:31.890496 28823 catalog_manager.cc:7201] Initiating ChangeConfig request to add master 127.20.1.187:37223: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "50669d11f58b4df087a03a1796667cb2"
member_type: NON_VOTER
last_known_addr {
host: "127.20.1.187"
port: 37223
}
attrs {
promote: true
}
}
dest_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e"
cas_config_opid_index: -1
I20260620 02:00:31.892066 28823 consensus_queue.cc:237] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: true } }
I20260620 02:00:31.895673 28907 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 1 FOLLOWER]: Refusing update from remote peer c110337d7e3b4d4db476c1ad3fd5e40e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 02:00:31.896003 28988 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 1 FOLLOWER]: Refusing update from remote peer c110337d7e3b4d4db476c1ad3fd5e40e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 02:00:31.896924 29325 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [LEADER]: Connected to new peer: Peer: permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 02:00:31.897548 29022 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260620 02:00:31.897920 28808 consensus_peers.cc:597] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e -> Peer 50669d11f58b4df087a03a1796667cb2 (127.20.1.187:37223): Couldn't send request to peer 50669d11f58b4df087a03a1796667cb2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260620 02:00:31.901835 29022 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 50669d11f58b4df087a03a1796667cb2 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: true } } }
I20260620 02:00:31.903384 28988 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 50669d11f58b4df087a03a1796667cb2 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: true } } }
I20260620 02:00:31.908063 29022 catalog_manager.cc:7205] Successfully completed master ChangeConfig request to add master 127.20.1.187:37223
I20260620 02:00:31.909127 29225 master.cc:562] Master@127.20.1.187:37223 shutting down...
I20260620 02:00:31.916396 29330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: true } } }
I20260620 02:00:31.916847 29330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:31.916276 28907 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 50669d11f58b4df087a03a1796667cb2 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: true } } }
I20260620 02:00:31.927294 29331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: true } } }
I20260620 02:00:31.927845 29331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:31.933787 29225 master.cc:584] Master@127.20.1.187:37223 shutdown complete.
I20260620 02:00:31.933938 29225 master_runner.cc:306] Clearing existing system tablet
I20260620 02:00:31.934511 29225 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:31.934636 29225 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:31.935842 29325 sys_catalog.cc:455] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: true } } }
I20260620 02:00:31.936213 29325 sys_catalog.cc:458] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:31.936977 29325 sys_catalog.cc:455] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: true } } }
I20260620 02:00:31.937207 29325 sys_catalog.cc:458] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:31.937551 29325 sys_catalog.cc:455] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: true } } }
I20260620 02:00:31.937727 29325 sys_catalog.cc:458] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:31.937916 29225 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 02:00:31.941888 29334 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:31.942651 29225 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:31.942973 29225 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:00:31.943325 29225 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "50669d11f58b4df087a03a1796667cb2"
format_stamp: "Formatted at 2026-06-20 02:00:31 on dist-test-slave-wg14"
I20260620 02:00:31.947615 29225 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:00:31.953763 29225 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 02:00:31.954022 29225 log.cc:1199] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: Deleting WAL directory at /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260620 02:00:31.954504 29225 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: Deleting consensus metadata
I20260620 02:00:31.954774 29225 master_runner.cc:316] Copying system tablet from 127.20.1.190:33851
I20260620 02:00:31.958544 29225 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.1.190:33851
I20260620 02:00:31.979995 28843 tablet_copy_service.cc:140] P c110337d7e3b4d4db476c1ad3fd5e40e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 50669d11f58b4df087a03a1796667cb2 ({username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:33648)
I20260620 02:00:31.980367 28843 tablet_copy_service.cc:161] P c110337d7e3b4d4db476c1ad3fd5e40e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 50669d11f58b4df087a03a1796667cb2 at {username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:33648: session id = 50669d11f58b4df087a03a1796667cb2-00000000000000000000000000000000
I20260620 02:00:31.984773 28843 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 02:00:31.989037 29225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 02:00:31.994741 29225 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: tablet copy: Starting download of 0 data blocks...
I20260620 02:00:31.995121 29225 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: tablet copy: Starting download of 1 WAL segments...
I20260620 02:00:32.001026 29225 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 02:00:32.007897 28843 tablet_copy_service.cc:342] P c110337d7e3b4d4db476c1ad3fd5e40e: Request end of tablet copy session 50669d11f58b4df087a03a1796667cb2-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:33648
I20260620 02:00:32.008221 28843 tablet_copy_service.cc:434] P c110337d7e3b4d4db476c1ad3fd5e40e: ending tablet copy session 50669d11f58b4df087a03a1796667cb2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 50669d11f58b4df087a03a1796667cb2
W20260620 02:00:32.016405 29225 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260620 02:00:32.025712 29225 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:32.029850 29342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:32.029892 29341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:32.030555 29344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:32.033793 29225 server_base.cc:1061] running on GCE node
I20260620 02:00:32.034448 29225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:32.035501 29225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:32.036736 29225 hybrid_clock.cc:648] HybridClock initialized: now 1781920832036629 us; error 73 us; skew 500 ppm
I20260620 02:00:32.037154 29225 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:32 dist-test-slave-wg14 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920832, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:32.039359 29225 init.cc:377] Logged in from keytab as kudu/127.20.1.187@KRBTEST.COM (short username kudu)
I20260620 02:00:32.043463 29225 webserver.cc:533] Webserver started at http://127.20.1.187:38605/ using document root <none> and password file <none>
I20260620 02:00:32.043906 29225 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:32.044009 29225 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:32.046887 29225 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 02:00:32.048908 29351 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:32.049376 29225 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 02:00:32.049516 29225 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "50669d11f58b4df087a03a1796667cb2"
format_stamp: "Formatted at 2026-06-20 02:00:31 on dist-test-slave-wg14"
I20260620 02:00:32.049695 29225 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:32.079226 29225 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:32.083390 29225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:32.116024 29225 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:37223
I20260620 02:00:32.116096 29402 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:37223 every 8 connection(s)
I20260620 02:00:32.120374 29225 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 02:00:32.129012 29403 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:00:32.132380 29403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: Bootstrap starting.
I20260620 02:00:32.163028 29403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:00:32.165421 29403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: Bootstrap complete.
I20260620 02:00:32.167768 29403 raft_consensus.cc:359] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: true } }
I20260620 02:00:32.168185 29403 raft_consensus.cc:740] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 50669d11f58b4df087a03a1796667cb2, State: Initialized, Role: LEARNER
I20260620 02:00:32.168495 29403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: true } }
I20260620 02:00:32.169539 29407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: true } } }
I20260620 02:00:32.169857 29403 sys_catalog.cc:565] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:00:32.170022 29407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:00:32.188812 29418 catalog_manager.cc:1294] Loaded cluster ID: 019cac4944f7460dbebf1976090a6846
I20260620 02:00:32.188946 29418 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: loading cluster ID for follower catalog manager: success
I20260620 02:00:32.193241 29418 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: acquiring CA information for follower catalog manager: success
I20260620 02:00:32.197531 29418 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:00:32.208050 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29080
Jun 20 02:00:32 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:00:32.243796 29205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:00:32.357610 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 29080
I20260620 02:00:32.380240 29378 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260620 02:00:32.382124 29407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c110337d7e3b4d4db476c1ad3fd5e40e. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: true } } }
I20260620 02:00:32.382447 29407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:00:32.383841 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:45257
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651
--webserver_port=40607
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651,127.20.1.187:37223 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:00:32.752264 29022 sys_catalog.cc:455] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: true } } }
I20260620 02:00:32.752846 29022 sys_catalog.cc:458] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:32.761580 29422 raft_consensus.cc:1064] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e: attempting to promote NON_VOTER 50669d11f58b4df087a03a1796667cb2 to VOTER
I20260620 02:00:32.764199 29422 consensus_queue.cc:237] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } }
I20260620 02:00:32.770357 29378 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [term 1 LEARNER]: Refusing update from remote peer c110337d7e3b4d4db476c1ad3fd5e40e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:00:32.770488 28988 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 1 FOLLOWER]: Refusing update from remote peer c110337d7e3b4d4db476c1ad3fd5e40e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:00:32.771692 28907 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 1 FOLLOWER]: Refusing update from remote peer c110337d7e3b4d4db476c1ad3fd5e40e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:00:32.771932 29022 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:00:32.774602 29022 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [LEADER]: Connected to new peer: Peer: permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:00:32.779361 29022 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [LEADER]: Connected to new peer: Peer: permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:00:32.780578 29423 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 50669d11f58b4df087a03a1796667cb2 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:32.790409 28907 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 50669d11f58b4df087a03a1796667cb2 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:32.800449 29378 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 50669d11f58b4df087a03a1796667cb2 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:32.804795 29426 sys_catalog.cc:455] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:32.805730 28988 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 50669d11f58b4df087a03a1796667cb2 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:32.805634 29022 sys_catalog.cc:455] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:32.806731 29022 sys_catalog.cc:458] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:32.807523 29022 sys_catalog.cc:455] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:32.807749 29022 sys_catalog.cc:458] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:32.806005 29423 sys_catalog.cc:455] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:32.808647 29423 sys_catalog.cc:458] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:32.808492 29426 sys_catalog.cc:458] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:32.798902 29428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:32.809527 29428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:32.818471 29407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:32.818825 29407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:32.821659 29429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:32.822202 29429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 02:00:32.928066 29421 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:32.928539 29421 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:32.928721 29421 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:32.938681 29421 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:32.938937 29421 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:00:32.956133 29421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:45257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=40607
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651,127.20.1.187:37223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:32.957866 29421 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:32.959447 29421 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:32.974567 29434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:32.974659 29435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:32.975158 29437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:32.975874 29421 server_base.cc:1061] running on GCE node
I20260620 02:00:32.976811 29421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:32.978462 29421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:32.980209 29421 hybrid_clock.cc:648] HybridClock initialized: now 1781920832980068 us; error 68 us; skew 500 ppm
I20260620 02:00:32.980643 29421 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:32 dist-test-slave-wg14 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920832, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:32.987406 29421 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:00:32.989850 29421 webserver.cc:533] Webserver started at http://127.20.1.129:40607/ using document root <none> and password file <none>
I20260620 02:00:32.990831 29421 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:32.991006 29421 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:33.002471 29421 fs_manager.cc:714] Time spent opening directory manager: real 0.009s user 0.003s sys 0.004s
I20260620 02:00:33.007756 29444 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:33.010455 29421 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s
I20260620 02:00:33.010731 29421 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "127c5477390c4c31b6f6375e5b1c6e68"
format_stamp: "Formatted at 2026-06-20 02:00:30 on dist-test-slave-wg14"
I20260620 02:00:33.011786 29421 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:33.038326 29421 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:33.043761 29421 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:33.044348 29421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:33.045620 29421 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:00:33.051522 29421 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:00:33.051741 29421 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.004s sys 0.000s
I20260620 02:00:33.051872 29421 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:00:33.051959 29421 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:33.099965 29556 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:45257 every 8 connection(s)
I20260620 02:00:33.100001 29421 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:45257
I20260620 02:00:33.102947 29421 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:00:33.110553 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 29421
Jun 20 02:00:33 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920832, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 02:00:33 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920832, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 02:00:33 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920832, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 02:00:33 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920832, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
I20260620 02:00:33.179402 29560 heartbeater.cc:344] Connected to a master server at 127.20.1.189:34413
I20260620 02:00:33.179422 29559 heartbeater.cc:344] Connected to a master server at 127.20.1.188:34651
I20260620 02:00:33.180764 29559 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:33.180764 29560 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:33.185210 29559 heartbeater.cc:507] Master 127.20.1.188:34651 requested a full tablet report, sending...
I20260620 02:00:33.186364 29558 heartbeater.cc:344] Connected to a master server at 127.20.1.190:33851
I20260620 02:00:33.185220 29560 heartbeater.cc:507] Master 127.20.1.189:34413 requested a full tablet report, sending...
I20260620 02:00:33.186954 29558 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:33.188061 29558 heartbeater.cc:507] Master 127.20.1.190:33851 requested a full tablet report, sending...
I20260620 02:00:33.189211 29557 heartbeater.cc:344] Connected to a master server at 127.20.1.187:37223
I20260620 02:00:33.189391 29557 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:33.189909 29557 heartbeater.cc:507] Master 127.20.1.187:37223 requested a full tablet report, sending...
I20260620 02:00:33.189965 28899 ts_manager.cc:194] Re-registered known tserver with Master: 127c5477390c4c31b6f6375e5b1c6e68 (127.20.1.129:45257)
I20260620 02:00:33.190311 28823 ts_manager.cc:194] Re-registered known tserver with Master: 127c5477390c4c31b6f6375e5b1c6e68 (127.20.1.129:45257)
I20260620 02:00:33.191319 28978 ts_manager.cc:194] Re-registered known tserver with Master: 127c5477390c4c31b6f6375e5b1c6e68 (127.20.1.129:45257)
I20260620 02:00:33.191444 28823 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:46607
I20260620 02:00:33.191741 29367 ts_manager.cc:194] Registered new tserver with Master: 127c5477390c4c31b6f6375e5b1c6e68 (127.20.1.129:45257)
I20260620 02:00:33.205448 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 02:00:33.225836 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 02:00:33.226125 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 28791
I20260620 02:00:33.227715 28853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:00:33.391183 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 28791
W20260620 02:00:33.437480 28962 connection.cc:573] server connection from 127.0.0.1:52908 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:00:33.438913 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:33851
--webserver_interface=127.20.1.190
--webserver_port=46711
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651
--master_addresses=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651,127.20.1.187:37223 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:00:33.971658 29571 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:33.972082 29571 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:33.972252 29571 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:33.983032 29571 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:00:33.983222 29571 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:33.983289 29571 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:00:33.983340 29571 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:00:33.999907 29571 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651,127.20.1.187:37223
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:33851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=46711
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:34.005340 29571 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:34.007572 29571 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:34.025027 29576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:34.027117 29577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:34.037561 29579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:34.039922 29571 server_base.cc:1061] running on GCE node
I20260620 02:00:34.041060 29571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:34.043565 29571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:34.052186 29571 hybrid_clock.cc:648] HybridClock initialized: now 1781920834051667 us; error 383 us; skew 500 ppm
I20260620 02:00:34.053184 29571 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:34 dist-test-slave-wg14 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920834, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:34.058579 29571 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:00:34.061638 29571 webserver.cc:533] Webserver started at http://127.20.1.190:46711/ using document root <none> and password file <none>
I20260620 02:00:34.062641 29571 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:34.062814 29571 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:34.072615 29571 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 02:00:34.078714 29586 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:34.081720 29571 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260620 02:00:34.081971 29571 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "c110337d7e3b4d4db476c1ad3fd5e40e"
format_stamp: "Formatted at 2026-06-20 02:00:27 on dist-test-slave-wg14"
I20260620 02:00:34.083084 29571 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:34.126204 29571 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:34.135056 29571 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:34.137557 29571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:34.181417 29571 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:33851
I20260620 02:00:34.181437 29637 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:33851 every 8 connection(s)
I20260620 02:00:34.183601 29571 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:00:34.190222 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 29571
I20260620 02:00:34.190608 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29421
I20260620 02:00:34.194881 29638 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:00:34.197049 29558 heartbeater.cc:499] Master 127.20.1.190:33851 was elected leader, sending a full tablet report...
I20260620 02:00:34.207785 29638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e: Bootstrap starting.
I20260620 02:00:34.258855 29638 log.cc:826] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:00:34.267833 29552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:00:34.307775 29638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:00:34.308650 29638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e: Bootstrap complete.
I20260620 02:00:34.321682 29648 raft_consensus.cc:493] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c110337d7e3b4d4db476c1ad3fd5e40e)
I20260620 02:00:34.322328 29648 raft_consensus.cc:515] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } }
I20260620 02:00:34.324828 29638 raft_consensus.cc:359] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } }
I20260620 02:00:34.327551 29638 raft_consensus.cc:740] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c110337d7e3b4d4db476c1ad3fd5e40e, State: Initialized, Role: FOLLOWER
I20260620 02:00:34.328755 29638 consensus_queue.cc:260] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } }
I20260620 02:00:34.331305 28944 master_runner.cc:187] Error getting master registration for 127.20.1.190:33851: OK, instance_id { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" instance_seqno: 1781920834163694 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:00:34.331753 28944 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:00:34.335954 29652 sys_catalog.cc:455] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:34.336513 29652 sys_catalog.cc:458] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:34.340107 29638 sys_catalog.cc:565] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:00:34.341269 29648 leader_election.cc:290] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c110337d7e3b4d4db476c1ad3fd5e40e (127.20.1.190:33851), 429d9d90eb8f4700a2adcd38da86dc6e (127.20.1.189:34413), 9aa09029e9694539a419a147fd3a9e15 (127.20.1.188:34651)
Jun 20 02:00:34 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920832, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:00:34.355161 28907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50669d11f58b4df087a03a1796667cb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" is_pre_election: true
I20260620 02:00:34.366739 28988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50669d11f58b4df087a03a1796667cb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "9aa09029e9694539a419a147fd3a9e15" is_pre_election: true
I20260620 02:00:34.368076 29354 leader_election.cc:304] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 50669d11f58b4df087a03a1796667cb2; no voters: 429d9d90eb8f4700a2adcd38da86dc6e, 9aa09029e9694539a419a147fd3a9e15
I20260620 02:00:34.368647 29648 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:00:34.371671 29613 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50669d11f58b4df087a03a1796667cb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" is_pre_election: true
I20260620 02:00:34.372473 29613 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50669d11f58b4df087a03a1796667cb2 in term 1.
I20260620 02:00:34.393023 29665 catalog_manager.cc:1294] Loaded cluster ID: 019cac4944f7460dbebf1976090a6846
I20260620 02:00:34.393198 29665 catalog_manager.cc:1587] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e: loading cluster ID for follower catalog manager: success
I20260620 02:00:34.402830 29665 catalog_manager.cc:1609] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e: acquiring CA information for follower catalog manager: success
Jun 20 02:00:34 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920834, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:00:34.410097 29665 catalog_manager.cc:1637] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:00:34.471894 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 29421
I20260620 02:00:34.480257 29571 master_runner.cc:264] No leader master found from master c110337d7e3b4d4db476c1ad3fd5e40e
I20260620 02:00:34.480477 29571 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:00:34.497896 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:45257
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651
--webserver_port=40607
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651,127.20.1.187:37223 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:00:34.504268 29643 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:34.201950 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:43119 (local address 127.20.1.190:33851)
0620 02:00:34.211289 (+ 9339us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:34.211310 (+ 21us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:34.218274 (+ 6964us) server_negotiation.cc:408] Connection header received
0620 02:00:34.222854 (+ 4580us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:00:34.222890 (+ 36us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:00:34.223548 (+ 658us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:00:34.225470 (+ 1922us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:00:34.227933 (+ 2463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 02:00:34.239543 (+ 11610us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:00:34.243614 (+ 4071us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 02:00:34.255354 (+ 11740us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0620 02:00:34.262208 (+ 6854us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0620 02:00:34.262319 (+ 111us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0620 02:00:34.262348 (+ 29us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0620 02:00:34.262406 (+ 58us) server_negotiation.cc:923] calling sasl_server_start()
0620 02:00:34.296499 (+ 34093us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0620 02:00:34.471818 (+175319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0620 02:00:34.471827 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0620 02:00:34.471831 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0620 02:00:34.472000 (+ 169us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0620 02:00:34.497246 (+ 25246us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:43119: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":8983,"thread_start_us":13963,"threads_started":1}
I20260620 02:00:34.891436 29670 raft_consensus.cc:493] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c110337d7e3b4d4db476c1ad3fd5e40e)
I20260620 02:00:34.891778 29670 raft_consensus.cc:515] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } }
I20260620 02:00:34.903246 29670 leader_election.cc:290] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c110337d7e3b4d4db476c1ad3fd5e40e (127.20.1.190:33851), 9aa09029e9694539a419a147fd3a9e15 (127.20.1.188:34651), 50669d11f58b4df087a03a1796667cb2 (127.20.1.187:37223)
I20260620 02:00:34.909868 29671 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c110337d7e3b4d4db476c1ad3fd5e40e)
I20260620 02:00:34.911482 29671 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } }
I20260620 02:00:34.914564 28988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "9aa09029e9694539a419a147fd3a9e15" is_pre_election: true
I20260620 02:00:34.915019 28988 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 429d9d90eb8f4700a2adcd38da86dc6e in term 1.
I20260620 02:00:34.916714 29613 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" is_pre_election: true
I20260620 02:00:34.917122 29613 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 429d9d90eb8f4700a2adcd38da86dc6e in term 1.
I20260620 02:00:34.918251 28885 leader_election.cc:304] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 429d9d90eb8f4700a2adcd38da86dc6e, 9aa09029e9694539a419a147fd3a9e15, c110337d7e3b4d4db476c1ad3fd5e40e; no voters:
I20260620 02:00:34.919261 29613 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9aa09029e9694539a419a147fd3a9e15" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" is_pre_election: true
I20260620 02:00:34.919615 29613 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aa09029e9694539a419a147fd3a9e15 in term 1.
I20260620 02:00:34.920614 29670 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 02:00:34.920804 29670 raft_consensus.cc:493] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 1 FOLLOWER]: Starting leader election (detected failure of leader c110337d7e3b4d4db476c1ad3fd5e40e)
I20260620 02:00:34.920924 29670 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:00:34.926339 29670 raft_consensus.cc:515] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } }
I20260620 02:00:34.928982 29613 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e"
I20260620 02:00:34.929325 29613 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:00:34.932780 29670 leader_election.cc:290] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [CANDIDATE]: Term 2 election: Requested vote from peers c110337d7e3b4d4db476c1ad3fd5e40e (127.20.1.190:33851), 9aa09029e9694539a419a147fd3a9e15 (127.20.1.188:34651), 50669d11f58b4df087a03a1796667cb2 (127.20.1.187:37223)
I20260620 02:00:34.933565 29671 leader_election.cc:290] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c110337d7e3b4d4db476c1ad3fd5e40e (127.20.1.190:33851), 429d9d90eb8f4700a2adcd38da86dc6e (127.20.1.189:34413), 50669d11f58b4df087a03a1796667cb2 (127.20.1.187:37223)
I20260620 02:00:34.933894 28988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "9aa09029e9694539a419a147fd3a9e15"
I20260620 02:00:34.934232 28988 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:00:34.934386 28907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9aa09029e9694539a419a147fd3a9e15" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" is_pre_election: true
I20260620 02:00:34.934876 28907 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9aa09029e9694539a419a147fd3a9e15 in current term 2: Already voted for candidate 429d9d90eb8f4700a2adcd38da86dc6e in this term.
I20260620 02:00:34.937492 28988 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 429d9d90eb8f4700a2adcd38da86dc6e in term 2.
I20260620 02:00:34.938212 29613 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 429d9d90eb8f4700a2adcd38da86dc6e in term 2.
I20260620 02:00:34.939275 29377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "50669d11f58b4df087a03a1796667cb2"
I20260620 02:00:34.939818 29377 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:00:34.939819 29378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "50669d11f58b4df087a03a1796667cb2" is_pre_election: true
I20260620 02:00:34.947904 29377 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 429d9d90eb8f4700a2adcd38da86dc6e in term 2.
I20260620 02:00:34.948707 28884 leader_election.cc:304] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 429d9d90eb8f4700a2adcd38da86dc6e, 50669d11f58b4df087a03a1796667cb2, c110337d7e3b4d4db476c1ad3fd5e40e; no voters:
I20260620 02:00:34.949311 29670 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:00:34.958735 29670 raft_consensus.cc:697] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [term 2 LEADER]: Becoming Leader. State: Replica: 429d9d90eb8f4700a2adcd38da86dc6e, State: Running, Role: LEADER
I20260620 02:00:34.959498 29670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } }
I20260620 02:00:34.959923 29377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9aa09029e9694539a419a147fd3a9e15" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "50669d11f58b4df087a03a1796667cb2" is_pre_election: true
I20260620 02:00:34.960359 29377 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9aa09029e9694539a419a147fd3a9e15 in current term 2: Already voted for candidate 429d9d90eb8f4700a2adcd38da86dc6e in this term.
I20260620 02:00:34.961270 28963 leader_election.cc:304] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 9aa09029e9694539a419a147fd3a9e15, c110337d7e3b4d4db476c1ad3fd5e40e; no voters: 429d9d90eb8f4700a2adcd38da86dc6e, 50669d11f58b4df087a03a1796667cb2
I20260620 02:00:34.961791 29671 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:00:34.965116 29675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 429d9d90eb8f4700a2adcd38da86dc6e. Latest consensus state: current_term: 2 leader_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:34.965454 29675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:34.966287 29677 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:00:34.969981 29677 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:00:34.972049 29677 catalog_manager.cc:1294] Loaded cluster ID: 019cac4944f7460dbebf1976090a6846
I20260620 02:00:34.972188 29677 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:00:34.973973 29677 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:00:34.975544 29677 catalog_manager.cc:6080] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e: Loaded TSK: 0
I20260620 02:00:34.976847 29677 catalog_manager.cc:1549] Initializing in-progress tserver states...
W20260620 02:00:35.021806 29669 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:35.022399 29669 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:35.022575 29669 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:35.034898 29669 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:35.035220 29669 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:00:35.051450 29669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:45257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=40607
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651,127.20.1.187:37223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:35.055352 29669 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:35.057679 29669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:35.073706 29683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:35.073581 29682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:35.074492 29685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:35.076222 29669 server_base.cc:1061] running on GCE node
I20260620 02:00:35.077024 29669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:35.079643 29669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:35.081133 29669 hybrid_clock.cc:648] HybridClock initialized: now 1781920835080970 us; error 109 us; skew 500 ppm
I20260620 02:00:35.081748 29669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:35 dist-test-slave-wg14 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920835, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:35.086575 29669 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:00:35.089534 29669 webserver.cc:533] Webserver started at http://127.20.1.129:40607/ using document root <none> and password file <none>
I20260620 02:00:35.090488 29669 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:35.090648 29669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:35.099025 29669 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:00:35.104979 29692 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:35.107975 29669 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.002s
I20260620 02:00:35.108282 29669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "127c5477390c4c31b6f6375e5b1c6e68"
format_stamp: "Formatted at 2026-06-20 02:00:30 on dist-test-slave-wg14"
I20260620 02:00:35.109397 29669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:35.135645 29669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:35.142016 29669 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:35.142318 29669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:35.144994 29669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:00:35.148240 29669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:00:35.148396 29669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:35.148511 29669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:00:35.148583 29669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:35.216243 29669 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:45257
I20260620 02:00:35.216274 29804 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:45257 every 8 connection(s)
I20260620 02:00:35.219269 29669 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:00:35.220976 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 29669
Jun 20 02:00:35 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920835, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
I20260620 02:00:35.263715 29805 heartbeater.cc:344] Connected to a master server at 127.20.1.187:37223
I20260620 02:00:35.264400 29805 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:35.265666 29805 heartbeater.cc:507] Master 127.20.1.187:37223 requested a full tablet report, sending...
Jun 20 02:00:35 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920835, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:00:35.270274 29367 ts_manager.cc:194] Re-registered known tserver with Master: 127c5477390c4c31b6f6375e5b1c6e68 (127.20.1.129:45257)
Jun 20 02:00:35 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920835, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 02:00:35 dist-test-slave-wg14 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920835, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:00:35.285704 29806 heartbeater.cc:344] Connected to a master server at 127.20.1.190:33851
I20260620 02:00:35.285941 29806 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:35.286988 29806 heartbeater.cc:507] Master 127.20.1.190:33851 requested a full tablet report, sending...
I20260620 02:00:35.289891 29603 ts_manager.cc:194] Registered new tserver with Master: 127c5477390c4c31b6f6375e5b1c6e68 (127.20.1.129:45257)
I20260620 02:00:35.296036 29810 heartbeater.cc:344] Connected to a master server at 127.20.1.189:34413
I20260620 02:00:35.296247 29810 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:35.296859 29810 heartbeater.cc:507] Master 127.20.1.189:34413 requested a full tablet report, sending...
I20260620 02:00:35.297227 29807 heartbeater.cc:344] Connected to a master server at 127.20.1.188:34651
I20260620 02:00:35.297420 29807 heartbeater.cc:461] Registering TS with master...
I20260620 02:00:35.298280 29807 heartbeater.cc:507] Master 127.20.1.188:34651 requested a full tablet report, sending...
I20260620 02:00:35.298419 28899 ts_manager.cc:194] Re-registered known tserver with Master: 127c5477390c4c31b6f6375e5b1c6e68 (127.20.1.129:45257)
I20260620 02:00:35.300478 28978 ts_manager.cc:194] Re-registered known tserver with Master: 127c5477390c4c31b6f6375e5b1c6e68 (127.20.1.129:45257)
I20260620 02:00:35.301162 28899 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:57163
I20260620 02:00:35.316255 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29669
W20260620 02:00:35.317749 29819 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:35.313348 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:49294 (local address 127.20.1.129:45257)
0620 02:00:35.314093 (+ 745us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:35.314100 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:35.314134 (+ 34us) server_negotiation.cc:408] Connection header received
0620 02:00:35.314226 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:00:35.314236 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:00:35.314531 (+ 295us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:00:35.314800 (+ 269us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:00:35.316055 (+ 1255us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49294: BlockingRecv error: recv got EOF from 127.0.0.1:49294 (error 108)
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":422,"threads_started":1}
W20260620 02:00:35.319325 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29669: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:45257: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:00:35.319468 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 29669
I20260620 02:00:35.336900 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:45257
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651
--webserver_port=40607
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651,127.20.1.187:37223
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:00:35.354065 29613 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [term 2 FOLLOWER]: Refusing update from remote peer 429d9d90eb8f4700a2adcd38da86dc6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260620 02:00:35.355393 29675 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 02:00:35.373194 29822 sys_catalog.cc:455] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 429d9d90eb8f4700a2adcd38da86dc6e. Latest consensus state: current_term: 2 leader_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:35.373644 29822 sys_catalog.cc:458] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:35.500568 29377 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [term 2 FOLLOWER]: Refusing update from remote peer 429d9d90eb8f4700a2adcd38da86dc6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260620 02:00:35.501456 29675 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 02:00:35.518956 29824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 429d9d90eb8f4700a2adcd38da86dc6e. Latest consensus state: current_term: 2 leader_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:35.519750 29824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:35.526268 28988 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [term 2 FOLLOWER]: Refusing update from remote peer 429d9d90eb8f4700a2adcd38da86dc6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260620 02:00:35.528090 29675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:35.528402 29675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:35.531589 29828 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 02:00:35.533308 29826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:35.533735 29826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: This master's current role is: LEADER
I20260620 02:00:35.535538 29822 sys_catalog.cc:455] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:35.535841 29822 sys_catalog.cc:458] T 00000000000000000000000000000000 P c110337d7e3b4d4db476c1ad3fd5e40e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:35.538040 29824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:35.538530 29824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50669d11f58b4df087a03a1796667cb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:35.539738 29835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:35.540019 29835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:00:35.539987 29832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 429d9d90eb8f4700a2adcd38da86dc6e. Latest consensus state: current_term: 2 leader_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:35.540225 29832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa09029e9694539a419a147fd3a9e15 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 02:00:35.814869 29821 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:35.815295 29821 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:35.815446 29821 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 02:00:35.815522 29821 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:35.828789 29821 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:35.829121 29821 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:00:35.842051 29821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:45257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=40607
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651,127.20.1.187:37223
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:35.844198 29821 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:35.845950 29821 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:35.861799 29841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:35.861896 29840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:35.862079 29843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:35.863603 29821 server_base.cc:1061] running on GCE node
I20260620 02:00:35.864444 29821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:35.866073 29821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:35.867911 29821 hybrid_clock.cc:648] HybridClock initialized: now 1781920835867618 us; error 163 us; skew 500 ppm
I20260620 02:00:35.868363 29821 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:35 dist-test-slave-wg14 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920835, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:35.872817 29821 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:00:35.875195 29821 webserver.cc:533] Webserver started at http://127.20.1.129:40607/ using document root <none> and password file <none>
I20260620 02:00:35.876288 29821 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:35.876407 29821 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:35.885138 29821 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.003s
I20260620 02:00:35.889506 29850 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:35.891618 29821 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 02:00:35.891850 29821 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "127c5477390c4c31b6f6375e5b1c6e68"
format_stamp: "Formatted at 2026-06-20 02:00:30 on dist-test-slave-wg14"
I20260620 02:00:35.892765 29821 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:35.911072 29821 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:35.916503 29821 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:35.916950 29821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:35.919427 29821 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:00:35.922768 29821 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:00:35.922984 29821 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:35.923178 29821 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:00:35.923481 29821 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.002s
W20260620 02:00:35.924532 29821 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 02:00:35.924690 29821 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 02:00:35.924765 29821 fault_injection.cc:59] injecting fault for kudu (pid 29821): FLAGS_fault_before_start (process will exit)
I20260620 02:00:35.948782 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:45257
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651
--webserver_port=40607
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651,127.20.1.187:37223 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:00:35.996227 29826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c110337d7e3b4d4db476c1ad3fd5e40e" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33851 } } peers { permanent_uuid: "429d9d90eb8f4700a2adcd38da86dc6e" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 34413 } } peers { permanent_uuid: "9aa09029e9694539a419a147fd3a9e15" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 34651 } } peers { permanent_uuid: "50669d11f58b4df087a03a1796667cb2" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37223 } attrs { promote: false } } }
I20260620 02:00:35.996567 29826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 429d9d90eb8f4700a2adcd38da86dc6e [sys.catalog]: This master's current role is: LEADER
W20260620 02:00:36.415392 29858 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:00:36.415786 29858 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:00:36.415897 29858 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:00:36.429652 29858 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:00:36.429926 29858 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:00:36.445420 29858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38915,127.20.1.149:45427,127.20.1.150:34359,127.20.1.151:44645,127.20.1.152:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:45257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=40607
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:33851,127.20.1.189:34413,127.20.1.188:34651,127.20.1.187:37223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:00:36.447685 29858 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:00:36.449585 29858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:00:36.464754 29863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:36.465055 29866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:00:36.465457 29864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:00:36.467739 29858 server_base.cc:1061] running on GCE node
I20260620 02:00:36.468568 29858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:00:36.470348 29858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:00:36.473870 29858 hybrid_clock.cc:648] HybridClock initialized: now 1781920836473650 us; error 154 us; skew 500 ppm
I20260620 02:00:36.474334 29858 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:00:36 dist-test-slave-wg14 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920836, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:00:36.481359 29858 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:00:36.485266 29858 webserver.cc:533] Webserver started at http://127.20.1.129:40607/ using document root <none> and password file <none>
I20260620 02:00:36.486212 29858 fs_manager.cc:362] Metadata directory not provided
I20260620 02:00:36.486340 29858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:00:36.495518 29858 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:00:36.499933 29873 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:36.502823 29858 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260620 02:00:36.503086 29858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "127c5477390c4c31b6f6375e5b1c6e68"
format_stamp: "Formatted at 2026-06-20 02:00:30 on dist-test-slave-wg14"
I20260620 02:00:36.504166 29858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:00:36.537027 29858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:00:36.545766 29858 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:00:36.546339 29858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:00:36.548240 29858 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:00:36.550468 29858 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:00:36.550628 29858 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:36.550793 29858 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:00:36.550868 29858 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:00:36.610385 29858 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:45257
I20260620 02:00:36.610419 29985 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:45257 every 8 connection(s)
I20260620 02:00:36.612288 29858 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:00:36.619877 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 29858
I20260620 02:00:36.620462 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29858
W20260620 02:00:36.659279 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29858: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:45257: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:00:36.659476 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 29858
W20260620 02:00:36.687057 29999 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:36.642174 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:51143 (local address 127.20.1.188:34651)
0620 02:00:36.642944 (+ 770us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:36.642952 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:36.658119 (+ 15167us) server_negotiation.cc:408] Connection header received
0620 02:00:36.658306 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:00:36.658312 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:00:36.658408 (+ 96us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:00:36.658536 (+ 128us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:00:36.685984 (+ 27448us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:51143: BlockingRecv error: recv got EOF from 127.20.1.129:51143 (error 108)
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":510,"threads_started":1}
W20260620 02:00:36.687057 29990 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:36.618085 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:59249 (local address 127.20.1.187:37223)
0620 02:00:36.626917 (+ 8832us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:36.626927 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:36.639051 (+ 12124us) server_negotiation.cc:408] Connection header received
0620 02:00:36.685984 (+ 46933us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:59249: BlockingRecv error: recv got EOF from 127.20.1.129:59249 (error 108)
Metrics: {"server-negotiator.queue_time_us":7664,"thread_start_us":7454,"threads_started":1}
W20260620 02:00:36.687177 29996 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:36.623572 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:56153 (local address 127.20.1.189:34413)
0620 02:00:36.643342 (+ 19770us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:36.643349 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:36.645593 (+ 2244us) server_negotiation.cc:408] Connection header received
0620 02:00:36.649865 (+ 4272us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:00:36.649873 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:00:36.650110 (+ 237us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:00:36.650513 (+ 403us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:00:36.655296 (+ 4783us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 02:00:36.656413 (+ 1117us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:00:36.658765 (+ 2352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 02:00:36.663032 (+ 4267us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0620 02:00:36.686070 (+ 23038us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:56153: BlockingRecv error: recv got EOF from 127.20.1.129:56153 (error 108)
Metrics: {"server-negotiator.queue_time_us":19436,"thread_start_us":19564,"threads_started":1}
I20260620 02:00:36.687824 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29571
W20260620 02:00:36.688568 29991 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:00:36.618301 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:56221 (local address 127.20.1.190:33851)
0620 02:00:36.645010 (+ 26709us) server_negotiation.cc:207] Beginning negotiation
0620 02:00:36.645018 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 02:00:36.645868 (+ 850us) server_negotiation.cc:408] Connection header received
0620 02:00:36.662525 (+ 16657us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:00:36.662532 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:00:36.662622 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:00:36.662755 (+ 133us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:00:36.688321 (+ 25566us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:56221: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":26548,"thread_start_us":26719,"threads_started":1}
W20260620 02:00:36.699481 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29571: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:33851: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:00:36.699695 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 29571
I20260620 02:00:36.725706 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 28867
I20260620 02:00:36.727470 28929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:00:36.873405 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 28867
I20260620 02:00:36.904150 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 28944
I20260620 02:00:36.905990 29008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:00:37.051990 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 28944
I20260620 02:00:37.084661 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 29225
I20260620 02:00:37.086673 29398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:00:37.283787 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 29225
2026-06-20T02:00:37Z chronyd exiting
2026-06-20T02:00:37Z chronyd exiting
2026-06-20T02:00:37Z chronyd exiting
2026-06-20T02:00:37Z chronyd exiting
2026-06-20T02:00:37Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/8 (14476 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/9
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:00:37 dist-test-slave-wg14 krb5kdc[30006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:00:37 dist-test-slave-wg14 krb5kdc[30006](info): set up 2 sockets
Jun 20 02:00:37 dist-test-slave-wg14 krb5kdc[30006](info): commencing operation
krb5kdc: starting...
W20260620 02:00:40.063832 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.688s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 02:00:40 dist-test-slave-wg14 krb5kdc[30006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
W20260620 02:00:40.442284 20486 mini_kdc.cc:268] Time spent creating service keytab for hive/127.0.0.1: real 0.103s user 0.000s sys 0.020s
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5.conf
2026-06-20 02:00:53: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Jun 20 02:00:56 dist-test-slave-wg14 krb5kdc[30006](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920856, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 02:01:06 dist-test-slave-wg14 krb5kdc[30006](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920866, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 02:01:06 dist-test-slave-wg14 krb5kdc[30006](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920866, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:01:08.208405 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.190: real 0.108s user 0.003s sys 0.018s
I20260620 02:01:08.209520 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42199
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:01:08.936697 30307 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:01:08.937619 30307 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:01:08.939035 30307 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:01:08.951414 30307 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:01:08.951629 30307 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:01:08.951702 30307 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:01:08.951751 30307 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:01:08.979561 30307 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:01:08.981920 30307 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:01:08.983819 30307 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:09.002888 30312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:09.003942 30313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:09.005620 30315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:09.007261 30307 server_base.cc:1061] running on GCE node
I20260620 02:01:09.008677 30307 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:01:09.008939 30307 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:01:09.009029 30307 hybrid_clock.cc:648] HybridClock initialized: now 1781920869009007 us; error 0 us; skew 500 ppm
I20260620 02:01:09.009425 30307 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:09 dist-test-slave-wg14 krb5kdc[30006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920869, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:09.019455 30307 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:01:09.022790 30307 webserver.cc:533] Webserver started at http://127.20.1.190:33757/ using document root <none> and password file <none>
I20260620 02:01:09.023901 30307 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:09.024416 30307 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:09.025393 30307 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:01:09.028644 30307 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "f3b6016fc44444139ac8878b81be5835"
format_stamp: "Formatted at 2026-06-20 02:01:09 on dist-test-slave-wg14"
I20260620 02:01:09.029745 30307 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "f3b6016fc44444139ac8878b81be5835"
format_stamp: "Formatted at 2026-06-20 02:01:09 on dist-test-slave-wg14"
I20260620 02:01:09.048049 30307 fs_manager.cc:696] Time spent creating directory manager: real 0.015s user 0.010s sys 0.004s
I20260620 02:01:09.061180 30321 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:09.063519 30307 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.005s sys 0.004s
I20260620 02:01:09.063764 30307 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "f3b6016fc44444139ac8878b81be5835"
format_stamp: "Formatted at 2026-06-20 02:01:09 on dist-test-slave-wg14"
I20260620 02:01:09.064009 30307 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:09.116577 30307 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:09.126091 30307 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:01:09.126389 30307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:09.174830 30307 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:42199
I20260620 02:01:09.174788 30372 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:42199 every 8 connection(s)
I20260620 02:01:09.176842 30307 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:01:09.181612 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 30307
I20260620 02:01:09.181849 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:01:09.185608 30373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:01:09.206624 30373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.189@KRBTEST.COM" created.
Jun 20 02:01:09 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920869, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:01:09.264709 30373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } has no permanent_uuid. Determining permanent_uuid...
W20260620 02:01:09.266955 30325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:42723: connect: Connection refused (error 111)
W20260620 02:01:09.272275 30373 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:42723: Network error: Client connection negotiation failed: client connection to 127.20.1.189:42723: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:01:09.306905 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.189: real 0.124s user 0.008s sys 0.027s
I20260620 02:01:09.322564 30373 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } attempt: 1
W20260620 02:01:09.325874 30373 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:42723: Network error: Client connection negotiation failed: client connection to 127.20.1.189:42723: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.189@KRBTEST.COM" created.
I20260620 02:01:09.394400 30373 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } attempt: 2
W20260620 02:01:09.399077 30373 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:42723: Network error: Client connection negotiation failed: client connection to 127.20.1.189:42723: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:01:09.401686 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:42723
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:01:09.490443 30373 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } attempt: 3
W20260620 02:01:09.496248 30373 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:42723: Network error: Client connection negotiation failed: client connection to 127.20.1.189:42723: connect: Connection refused (error 111)
I20260620 02:01:09.639690 30373 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } attempt: 4
W20260620 02:01:09.643779 30373 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:42723: Network error: Client connection negotiation failed: client connection to 127.20.1.189:42723: connect: Connection refused (error 111)
I20260620 02:01:09.943229 30373 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } attempt: 5
W20260620 02:01:09.946601 30373 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:42723: Network error: Client connection negotiation failed: client connection to 127.20.1.189:42723: connect: Connection refused (error 111)
W20260620 02:01:10.069260 30382 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:01:10.069837 30382 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:01:10.075490 30382 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:01:10.087251 30382 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:01:10.087401 30382 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:01:10.087474 30382 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:01:10.087554 30382 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:01:10.108713 30382 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:42723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:01:10.111410 30382 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:01:10.115975 30382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:10.135604 30388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:10.137972 30387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:10.140789 30390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:10.141734 30382 server_base.cc:1061] running on GCE node
I20260620 02:01:10.142848 30382 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:01:10.143074 30382 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:01:10.143206 30382 hybrid_clock.cc:648] HybridClock initialized: now 1781920870143177 us; error 0 us; skew 500 ppm
I20260620 02:01:10.143965 30382 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:10 dist-test-slave-wg14 krb5kdc[30006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920870, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:10.155512 30382 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 02:01:10.159006 30382 webserver.cc:533] Webserver started at http://127.20.1.189:42215/ using document root <none> and password file <none>
I20260620 02:01:10.159982 30382 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:10.160115 30382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:10.160503 30382 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:01:10.163679 30382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2"
format_stamp: "Formatted at 2026-06-20 02:01:10 on dist-test-slave-wg14"
I20260620 02:01:10.164736 30382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2"
format_stamp: "Formatted at 2026-06-20 02:01:10 on dist-test-slave-wg14"
I20260620 02:01:10.181412 30382 fs_manager.cc:696] Time spent creating directory manager: real 0.016s user 0.010s sys 0.001s
I20260620 02:01:10.186918 30396 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:10.189004 30382 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260620 02:01:10.189249 30382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2"
format_stamp: "Formatted at 2026-06-20 02:01:10 on dist-test-slave-wg14"
I20260620 02:01:10.189486 30382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:10.228924 30382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:10.236436 30382 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:01:10.237951 30382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:10.285265 30382 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:42723
I20260620 02:01:10.285323 30447 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:42723 every 8 connection(s)
I20260620 02:01:10.287333 30382 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 02:01:10.292133 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 30382
I20260620 02:01:10.292351 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 02:01:10.297979 30448 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:01:10.306443 30448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:01:10 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920870, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.188@KRBTEST.COM" created.
I20260620 02:01:10.358345 30448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:01:10 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920870, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:01:10.389046 30448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } has no permanent_uuid. Determining permanent_uuid...
W20260620 02:01:10.396669 30400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43391: connect: Connection refused (error 111)
W20260620 02:01:10.401201 30448 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43391: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43391: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:01:10.428471 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.188: real 0.136s user 0.006s sys 0.022s
I20260620 02:01:10.450724 30448 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } attempt: 1
W20260620 02:01:10.456528 30448 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43391: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43391: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.188@KRBTEST.COM" created.
I20260620 02:01:10.494057 30373 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } attempt: 6
Jun 20 02:01:10 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920869, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:01:10.524935 30448 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } attempt: 2
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:01:10.528052 30448 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43391: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43391: connect: Connection refused (error 111)
I20260620 02:01:10.528934 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:43391
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:01:10.539539 30373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } has no permanent_uuid. Determining permanent_uuid...
W20260620 02:01:10.543239 30373 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43391: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43391: connect: Connection refused (error 111)
I20260620 02:01:10.595681 30373 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } attempt: 1
W20260620 02:01:10.598297 30373 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43391: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43391: connect: Connection refused (error 111)
I20260620 02:01:10.619378 30448 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } attempt: 3
W20260620 02:01:10.622568 30448 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43391: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43391: connect: Connection refused (error 111)
I20260620 02:01:10.672668 30373 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } attempt: 2
W20260620 02:01:10.675421 30373 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43391: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43391: connect: Connection refused (error 111)
I20260620 02:01:10.765930 30448 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } attempt: 4
W20260620 02:01:10.770956 30448 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43391: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43391: connect: Connection refused (error 111)
I20260620 02:01:10.789968 30373 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } attempt: 3
W20260620 02:01:10.793128 30373 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43391: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43391: connect: Connection refused (error 111)
I20260620 02:01:10.942476 30373 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } attempt: 4
W20260620 02:01:10.946077 30373 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43391: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43391: connect: Connection refused (error 111)
I20260620 02:01:11.071424 30448 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } attempt: 5
W20260620 02:01:11.074859 30448 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43391: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43391: connect: Connection refused (error 111)
W20260620 02:01:11.140476 30459 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:01:11.141016 30459 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:01:11.141178 30459 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:01:11.151664 30459 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:01:11.152263 30459 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:01:11.152992 30459 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:01:11.153352 30459 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:01:11.168660 30459 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:43391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:01:11.171221 30459 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:01:11.172883 30459 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:11.186439 30465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:11.190132 30466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:11.199301 30468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:11.201592 30459 server_base.cc:1061] running on GCE node
I20260620 02:01:11.203035 30459 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:01:11.203361 30459 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:01:11.203486 30459 hybrid_clock.cc:648] HybridClock initialized: now 1781920871203456 us; error 0 us; skew 500 ppm
I20260620 02:01:11.204056 30459 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:11 dist-test-slave-wg14 krb5kdc[30006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920871, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:11.210570 30459 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 02:01:11.217559 30373 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } attempt: 5
I20260620 02:01:11.218499 30459 webserver.cc:533] Webserver started at http://127.20.1.188:41761/ using document root <none> and password file <none>
I20260620 02:01:11.219898 30459 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:11.220098 30459 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:11.220570 30459 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
W20260620 02:01:11.222177 30373 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:43391: Network error: Client connection negotiation failed: client connection to 127.20.1.188:43391: connect: Connection refused (error 111)
I20260620 02:01:11.225651 30459 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "4fff0de072014bd498aa189ff340dea7"
format_stamp: "Formatted at 2026-06-20 02:01:11 on dist-test-slave-wg14"
I20260620 02:01:11.228561 30459 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "4fff0de072014bd498aa189ff340dea7"
format_stamp: "Formatted at 2026-06-20 02:01:11 on dist-test-slave-wg14"
I20260620 02:01:11.237493 30459 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260620 02:01:11.242861 30474 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:11.244732 30459 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:01:11.245004 30459 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "4fff0de072014bd498aa189ff340dea7"
format_stamp: "Formatted at 2026-06-20 02:01:11 on dist-test-slave-wg14"
I20260620 02:01:11.245275 30459 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:11.284016 30459 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:11.290752 30459 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:01:11.291606 30459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:11.325953 30459 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:43391
I20260620 02:01:11.325976 30525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:43391 every 8 connection(s)
I20260620 02:01:11.327905 30459 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 02:01:11.334017 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 30459
I20260620 02:01:11.334473 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 02:01:11.335700 30526 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:01:11.348918 30526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:01:11 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 02:01:11 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920871, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 02:01:11 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:01:11.396950 30526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:01:11 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920871, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 02:01:11 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:01:11.429453 30526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:01:11 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920871, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:01:11.474678 30526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7: Bootstrap starting.
I20260620 02:01:11.482620 30526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7: Neither blocks nor log segments found. Creating new log.
I20260620 02:01:11.484023 30526 log.cc:826] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7: Log is configured to *not* fsync() on all Append() calls
I20260620 02:01:11.488739 30526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7: No bootstrap required, opened a new log
I20260620 02:01:11.504026 30526 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } }
I20260620 02:01:11.504513 30526 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:01:11.504657 30526 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fff0de072014bd498aa189ff340dea7, State: Initialized, Role: FOLLOWER
I20260620 02:01:11.505448 30526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } }
I20260620 02:01:11.506556 30537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } }
I20260620 02:01:11.506963 30537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:11.507882 30526 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:01:11.528141 30548 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:01:11.528299 30548 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:01:11.525463 30459 master_runner.cc:187] Error getting master registration for 127.20.1.190:42199: OK, instance_id { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" instance_seqno: 1781920869161872 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:01:11.533890 30459 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 02:01:11.545571 30480 tablet.cc:2406] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:01:11.622164 30448 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } attempt: 6
Jun 20 02:01:11 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920870, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:01:11.652563 30448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2: Bootstrap starting.
I20260620 02:01:11.665062 30448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2: Neither blocks nor log segments found. Creating new log.
I20260620 02:01:11.666903 30448 log.cc:826] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:01:11.675921 30448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2: No bootstrap required, opened a new log
I20260620 02:01:11.689904 30448 raft_consensus.cc:359] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } }
I20260620 02:01:11.691501 30448 raft_consensus.cc:385] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:01:11.691982 30448 raft_consensus.cc:740] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2ce249d2ef54e3c80bb6ca0c2e310f2, State: Initialized, Role: FOLLOWER
I20260620 02:01:11.694039 30448 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } }
I20260620 02:01:11.700644 30552 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } }
I20260620 02:01:11.701084 30552 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:11.702950 30448 sys_catalog.cc:565] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:01:11.729115 30563 catalog_manager.cc:1593] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:01:11.729374 30563 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:01:11.723210 30382 master_runner.cc:187] Error getting master registration for 127.20.1.190:42199: OK, instance_id { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" instance_seqno: 1781920869161872 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:01:11.739646 30382 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 02:01:11.746569 30402 tablet.cc:2406] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:01:11.761461 30373 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } attempt: 6
Jun 20 02:01:11 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920869, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:01:11.808470 30373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835: Bootstrap starting.
I20260620 02:01:11.815580 30373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835: Neither blocks nor log segments found. Creating new log.
I20260620 02:01:11.818938 30373 log.cc:826] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835: Log is configured to *not* fsync() on all Append() calls
I20260620 02:01:11.829833 30373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835: No bootstrap required, opened a new log
I20260620 02:01:11.840822 30373 raft_consensus.cc:359] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } }
I20260620 02:01:11.841367 30373 raft_consensus.cc:385] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:01:11.842247 30373 raft_consensus.cc:740] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b6016fc44444139ac8878b81be5835, State: Initialized, Role: FOLLOWER
I20260620 02:01:11.843516 30373 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } }
I20260620 02:01:11.845418 30567 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } }
I20260620 02:01:11.846031 30567 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:11.850422 30373 sys_catalog.cc:565] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:01:11.858714 30537 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:01:11.859411 30537 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } }
I20260620 02:01:11.863324 30348 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fff0de072014bd498aa189ff340dea7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3b6016fc44444139ac8878b81be5835" is_pre_election: true
I20260620 02:01:11.864457 30348 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fff0de072014bd498aa189ff340dea7 in term 0.
I20260620 02:01:11.865163 30423 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fff0de072014bd498aa189ff340dea7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" is_pre_election: true
I20260620 02:01:11.866446 30423 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fff0de072014bd498aa189ff340dea7 in term 0.
I20260620 02:01:11.868091 30478 leader_election.cc:304] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fff0de072014bd498aa189ff340dea7, f2ce249d2ef54e3c80bb6ca0c2e310f2; no voters:
I20260620 02:01:11.868911 30537 leader_election.cc:290] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3b6016fc44444139ac8878b81be5835 (127.20.1.190:42199), f2ce249d2ef54e3c80bb6ca0c2e310f2 (127.20.1.189:42723)
I20260620 02:01:11.869627 30537 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:01:11.869987 30537 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:01:11.870944 30537 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [term 0 FOLLOWER]: Advancing to term 1
W20260620 02:01:11.885551 30327 tablet.cc:2406] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:01:11.887395 30537 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } }
I20260620 02:01:11.892521 30348 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fff0de072014bd498aa189ff340dea7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3b6016fc44444139ac8878b81be5835"
I20260620 02:01:11.893693 30348 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:01:11.893987 30423 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fff0de072014bd498aa189ff340dea7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2"
I20260620 02:01:11.894585 30423 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:01:11.898856 30423 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fff0de072014bd498aa189ff340dea7 in term 1.
I20260620 02:01:11.899861 30478 leader_election.cc:304] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fff0de072014bd498aa189ff340dea7, f2ce249d2ef54e3c80bb6ca0c2e310f2; no voters:
I20260620 02:01:11.900321 30571 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:01:11.901427 30348 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fff0de072014bd498aa189ff340dea7 in term 1.
I20260620 02:01:11.908447 30537 leader_election.cc:290] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [CANDIDATE]: Term 1 election: Requested vote from peers f3b6016fc44444139ac8878b81be5835 (127.20.1.190:42199), f2ce249d2ef54e3c80bb6ca0c2e310f2 (127.20.1.189:42723)
I20260620 02:01:11.909922 30571 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [term 1 LEADER]: Becoming Leader. State: Replica: 4fff0de072014bd498aa189ff340dea7, State: Running, Role: LEADER
I20260620 02:01:11.910983 30571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } }
I20260620 02:01:11.931288 30537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fff0de072014bd498aa189ff340dea7. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } }
I20260620 02:01:11.932006 30537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:11.933604 30576 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:01:11.936511 30576 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:01:11.966255 30423 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [term 1 FOLLOWER]: Refusing update from remote peer 4fff0de072014bd498aa189ff340dea7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:01:11.969262 30348 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [term 1 FOLLOWER]: Refusing update from remote peer 4fff0de072014bd498aa189ff340dea7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:01:11.970806 30537 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 02:01:11.973557 30537 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:01:12.005579 30552 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fff0de072014bd498aa189ff340dea7. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } }
I20260620 02:01:12.006083 30552 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:12.019723 30537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } }
I20260620 02:01:12.020037 30537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:12.024315 30571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } }
I20260620 02:01:12.024689 30571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:12.027859 30576 catalog_manager.cc:1382] Generated new cluster ID: 0354708363be4f35a33f4b05654c434f
I20260620 02:01:12.028034 30576 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:01:12.020534 30567 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fff0de072014bd498aa189ff340dea7. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } }
I20260620 02:01:12.032086 30567 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:12.035708 30590 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } }
I20260620 02:01:12.041551 30584 mvcc.cc:204] Tried to move back new op lower bound from 7298747891528441856 to 7298747891355242496. Current Snapshot: MvccSnapshot[applied={T|T < 7298747891528441856 or (T in {7298747891528441856})}]
I20260620 02:01:12.042709 30552 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } }
I20260620 02:01:12.043282 30552 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:12.051419 30590 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:12.050415 30588 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:01:12.089421 30576 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:01:12.092059 30576 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:01:12.112287 30576 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7: Generated new TSK 0
I20260620 02:01:12.113970 30576 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:01:12.256052 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.131s user 0.004s sys 0.031s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:01:12.368650 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.112s user 0.002s sys 0.029s
I20260620 02:01:12.369601 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:01:12.732038 30563 catalog_manager.cc:1294] Loaded cluster ID: 0354708363be4f35a33f4b05654c434f
I20260620 02:01:12.732164 30563 catalog_manager.cc:1587] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2: loading cluster ID for follower catalog manager: success
I20260620 02:01:12.735733 30563 catalog_manager.cc:1609] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2: acquiring CA information for follower catalog manager: success
I20260620 02:01:12.738409 30563 catalog_manager.cc:1637] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 02:01:12.855341 30596 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:01:12.855997 30596 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:01:12.856173 30596 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:01:12.867662 30596 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:01:12.867925 30596 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:01:12.882107 30596 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:01:12.885588 30596 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:01:12.888599 30596 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:12.907531 30602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:12.911087 30601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:12.913952 30604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:12.914129 30596 server_base.cc:1061] running on GCE node
I20260620 02:01:12.915081 30596 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:01:12.915385 30596 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:01:12.915483 30596 hybrid_clock.cc:648] HybridClock initialized: now 1781920872915459 us; error 0 us; skew 500 ppm
I20260620 02:01:12.915839 30596 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:12 dist-test-slave-wg14 krb5kdc[30006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920872, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:12.920650 30596 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:01:12.925979 30596 webserver.cc:533] Webserver started at http://127.20.1.129:43585/ using document root <none> and password file <none>
I20260620 02:01:12.927302 30596 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:12.927486 30596 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:12.927951 30596 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:01:12.932545 30596 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "0fdcf25fbea94ad8a67f10a023127832"
format_stamp: "Formatted at 2026-06-20 02:01:12 on dist-test-slave-wg14"
I20260620 02:01:12.933828 30596 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "0fdcf25fbea94ad8a67f10a023127832"
format_stamp: "Formatted at 2026-06-20 02:01:12 on dist-test-slave-wg14"
I20260620 02:01:12.945573 30596 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.006s sys 0.003s
I20260620 02:01:12.952432 30610 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:12.954613 30596 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:01:12.954860 30596 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "0fdcf25fbea94ad8a67f10a023127832"
format_stamp: "Formatted at 2026-06-20 02:01:12 on dist-test-slave-wg14"
I20260620 02:01:12.955104 30596 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:12.996783 30596 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:13.007047 30596 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:01:13.007498 30596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:13.009624 30596 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:01:13.012183 30596 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:01:13.012301 30596 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:13.012423 30596 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:01:13.012485 30596 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:13.066702 30588 catalog_manager.cc:1294] Loaded cluster ID: 0354708363be4f35a33f4b05654c434f
I20260620 02:01:13.067170 30588 catalog_manager.cc:1587] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835: loading cluster ID for follower catalog manager: success
I20260620 02:01:13.075459 30588 catalog_manager.cc:1609] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835: acquiring CA information for follower catalog manager: success
I20260620 02:01:13.079820 30588 catalog_manager.cc:1637] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:01:13.095111 30596 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38373
I20260620 02:01:13.095268 30723 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38373 every 8 connection(s)
I20260620 02:01:13.097044 30596 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:01:13.106817 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 30596
I20260620 02:01:13.107050 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 02:01:13 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920872, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:01:13.161589 30725 heartbeater.cc:344] Connected to a master server at 127.20.1.190:42199
I20260620 02:01:13.162365 30725 heartbeater.cc:461] Registering TS with master...
I20260620 02:01:13.166426 30725 heartbeater.cc:507] Master 127.20.1.190:42199 requested a full tablet report, sending...
I20260620 02:01:13.172066 30338 ts_manager.cc:194] Registered new tserver with Master: 0fdcf25fbea94ad8a67f10a023127832 (127.20.1.129:38373)
Jun 20 02:01:13 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920872, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 02:01:13 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920872, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:01:13.206934 30724 heartbeater.cc:344] Connected to a master server at 127.20.1.188:43391
I20260620 02:01:13.207223 30724 heartbeater.cc:461] Registering TS with master...
I20260620 02:01:13.207912 30724 heartbeater.cc:507] Master 127.20.1.188:43391 requested a full tablet report, sending...
I20260620 02:01:13.213614 30491 ts_manager.cc:194] Registered new tserver with Master: 0fdcf25fbea94ad8a67f10a023127832 (127.20.1.129:38373)
I20260620 02:01:13.216547 30491 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:52577
I20260620 02:01:13.218149 30726 heartbeater.cc:344] Connected to a master server at 127.20.1.189:42723
I20260620 02:01:13.218631 30726 heartbeater.cc:461] Registering TS with master...
I20260620 02:01:13.219612 30726 heartbeater.cc:507] Master 127.20.1.189:42723 requested a full tablet report, sending...
I20260620 02:01:13.223532 30413 ts_manager.cc:194] Registered new tserver with Master: 0fdcf25fbea94ad8a67f10a023127832 (127.20.1.129:38373)
I20260620 02:01:13.229383 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
WARNING: no policy specified for kudu/127.20.1.187@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.187@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.187 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.187 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.187@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.187@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.187 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.187 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:01:13.429121 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:37653
--webserver_interface=127.20.1.187
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.187
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391,127.20.1.187:37653 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:01:13.959268 30737 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:01:13.959643 30737 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:01:13.959748 30737 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:01:13.974757 30737 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:01:13.974906 30737 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:01:13.974967 30737 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:01:13.975009 30737 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:01:13.992581 30737 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391,127.20.1.187:37653
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.187
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:37653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.20.1.187
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:01:13.994601 30737 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:01:13.996322 30737 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:14.009997 30743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:14.011973 30742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:14.012403 30745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:14.012799 30737 server_base.cc:1061] running on GCE node
I20260620 02:01:14.013996 30737 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:01:14.014430 30737 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:01:14.014540 30737 hybrid_clock.cc:648] HybridClock initialized: now 1781920874014518 us; error 0 us; skew 500 ppm
I20260620 02:01:14.014931 30737 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:14 dist-test-slave-wg14 krb5kdc[30006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920874, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:14.019380 30737 init.cc:377] Logged in from keytab as kudu/127.20.1.187@KRBTEST.COM (short username kudu)
I20260620 02:01:14.022769 30737 webserver.cc:533] Webserver started at http://127.20.1.187:46781/ using document root <none> and password file <none>
I20260620 02:01:14.024255 30737 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:14.024451 30737 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:14.024875 30737 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:01:14.028036 30737 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/data/instance:
uuid: "05111c362ece43ed8f2e37fccbab0594"
format_stamp: "Formatted at 2026-06-20 02:01:14 on dist-test-slave-wg14"
I20260620 02:01:14.029062 30737 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/wal/instance:
uuid: "05111c362ece43ed8f2e37fccbab0594"
format_stamp: "Formatted at 2026-06-20 02:01:14 on dist-test-slave-wg14"
I20260620 02:01:14.038467 30737 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.003s sys 0.005s
I20260620 02:01:14.045842 30751 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:14.050094 30737 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.005s sys 0.001s
I20260620 02:01:14.050714 30737 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "05111c362ece43ed8f2e37fccbab0594"
format_stamp: "Formatted at 2026-06-20 02:01:14 on dist-test-slave-wg14"
I20260620 02:01:14.051023 30737 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:14.076370 30737 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:14.083730 30737 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:01:14.084195 30737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:14.124626 30737 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:37653
I20260620 02:01:14.124830 30802 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:37653 every 8 connection(s)
I20260620 02:01:14.127123 30737 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 02:01:14.135859 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 30737
I20260620 02:01:14.138669 30803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:01:14.153856 30803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:01:14 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
Jun 20 02:01:14 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920874, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:01:14.200941 30803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:01:14 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920874, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:01:14.226440 30724 heartbeater.cc:499] Master 127.20.1.188:43391 was elected leader, sending a full tablet report...
I20260620 02:01:14.228497 30803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:01:14 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920874, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:01:14.252172 30803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:01:14 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920874, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
I20260620 02:01:14.290192 30803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: Bootstrap starting.
I20260620 02:01:14.295049 30803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: Neither blocks nor log segments found. Creating new log.
I20260620 02:01:14.296559 30803 log.cc:826] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: Log is configured to *not* fsync() on all Append() calls
I20260620 02:01:14.300397 30803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: No bootstrap required, opened a new log
I20260620 02:01:14.312048 30803 raft_consensus.cc:359] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } }
I20260620 02:01:14.312515 30803 raft_consensus.cc:385] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:01:14.312625 30803 raft_consensus.cc:740] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05111c362ece43ed8f2e37fccbab0594, State: Initialized, Role: FOLLOWER
I20260620 02:01:14.313549 30803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } }
I20260620 02:01:14.321377 30803 sys_catalog.cc:565] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:01:14.322409 30812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } } }
I20260620 02:01:14.323040 30812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 02:01:14.339078 30757 tablet.cc:2406] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:01:14.346351 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 30596
W20260620 02:01:14.350492 30823 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:01:14.350960 30823 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:01:14.372160 30737 master_runner.cc:429] Detected that this master 05111c362ece43ed8f2e37fccbab0594 is joining an existing cluster
I20260620 02:01:14.372368 30737 catalog_manager.cc:1024] Shutting down the Catalog Manager
Jun 20 02:01:14 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:01:14.378957 30737 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 02:01:14.379580 30737 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 02:01:14.379822 30737 tablet_replica.cc:333] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: stopping tablet replica
I20260620 02:01:14.384253 30737 master_runner.cc:434] Initiating AddMaster RPC to add 127.20.1.187:37653
I20260620 02:01:14.398711 30719 generic_service.cc:196] Checking for leaks (request via RPC)
Jun 20 02:01:14 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920871, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
I20260620 02:01:14.516664 30491 catalog_manager.cc:7201] Initiating ChangeConfig request to add master 127.20.1.187:37653: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "05111c362ece43ed8f2e37fccbab0594"
member_type: NON_VOTER
last_known_addr {
host: "127.20.1.187"
port: 37653
}
attrs {
promote: true
}
}
dest_uuid: "4fff0de072014bd498aa189ff340dea7"
cas_config_opid_index: -1
I20260620 02:01:14.519287 30491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: true } }
I20260620 02:01:14.524221 30348 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [term 1 FOLLOWER]: Refusing update from remote peer 4fff0de072014bd498aa189ff340dea7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 02:01:14.525267 30826 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260620 02:01:14.527877 30478 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 -> Peer 05111c362ece43ed8f2e37fccbab0594 (127.20.1.187:37653): Couldn't send request to peer 05111c362ece43ed8f2e37fccbab0594. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260620 02:01:14.542043 30422 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [term 1 FOLLOWER]: Refusing update from remote peer 4fff0de072014bd498aa189ff340dea7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 02:01:14.542742 30826 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 05111c362ece43ed8f2e37fccbab0594 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: true } } }
I20260620 02:01:14.544894 30348 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 05111c362ece43ed8f2e37fccbab0594 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: true } } }
I20260620 02:01:14.551051 30839 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 02:01:14.557726 30826 catalog_manager.cc:7205] Successfully completed master ChangeConfig request to add master 127.20.1.187:37653
I20260620 02:01:14.559033 30737 master.cc:562] Master@127.20.1.187:37653 shutting down...
I20260620 02:01:14.565093 30841 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: true } } }
I20260620 02:01:14.565493 30841 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:14.571024 30826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: true } } }
I20260620 02:01:14.572342 30826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:14.571890 30842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: true } } }
I20260620 02:01:14.572896 30842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:14.577214 30422 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 05111c362ece43ed8f2e37fccbab0594 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: true } } }
I20260620 02:01:14.587477 30847 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: true } } }
I20260620 02:01:14.588729 30847 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:14.613178 30737 master.cc:584] Master@127.20.1.187:37653 shutdown complete.
I20260620 02:01:14.613418 30737 master_runner.cc:306] Clearing existing system tablet
I20260620 02:01:14.613916 30737 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:14.614022 30737 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:14.615005 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 30596
I20260620 02:01:14.624979 30737 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.004s sys 0.004s
I20260620 02:01:14.629956 30849 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:14.631460 30737 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:14.631911 30737 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 02:01:14.632043 30737 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "05111c362ece43ed8f2e37fccbab0594"
format_stamp: "Formatted at 2026-06-20 02:01:14 on dist-test-slave-wg14"
I20260620 02:01:14.637617 30737 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:01:14.648679 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:38373
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391
--webserver_port=43585
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391,127.20.1.187:37653 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:01:14.649597 30737 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 02:01:14.649854 30737 log.cc:1199] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: Deleting WAL directory at /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260620 02:01:14.650655 30737 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: Deleting consensus metadata
I20260620 02:01:14.651229 30737 master_runner.cc:316] Copying system tablet from 127.20.1.188:43391
I20260620 02:01:14.670128 30737 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: tablet copy: Beginning tablet copy session from remote peer at address 127.20.1.188:43391
I20260620 02:01:14.716352 30511 tablet_copy_service.cc:140] P 4fff0de072014bd498aa189ff340dea7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 05111c362ece43ed8f2e37fccbab0594 ({username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:39576)
I20260620 02:01:14.716858 30511 tablet_copy_service.cc:161] P 4fff0de072014bd498aa189ff340dea7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 05111c362ece43ed8f2e37fccbab0594 at {username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:39576: session id = 05111c362ece43ed8f2e37fccbab0594-00000000000000000000000000000000
I20260620 02:01:14.722514 30511 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 02:01:14.727480 30737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 02:01:14.737906 30737 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: tablet copy: Starting download of 0 data blocks...
I20260620 02:01:14.738458 30737 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: tablet copy: Starting download of 1 WAL segments...
I20260620 02:01:14.744328 30737 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 02:01:14.752112 30511 tablet_copy_service.cc:342] P 4fff0de072014bd498aa189ff340dea7: Request end of tablet copy session 05111c362ece43ed8f2e37fccbab0594-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:39576
I20260620 02:01:14.752684 30511 tablet_copy_service.cc:434] P 4fff0de072014bd498aa189ff340dea7: ending tablet copy session 05111c362ece43ed8f2e37fccbab0594-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 05111c362ece43ed8f2e37fccbab0594
I20260620 02:01:14.772953 30737 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:14.784332 30857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:14.786697 30858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:14.793439 30861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:14.796959 30737 server_base.cc:1061] running on GCE node
I20260620 02:01:14.797344 30737 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:01:14.797422 30737 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:01:14.797499 30737 hybrid_clock.cc:648] HybridClock initialized: now 1781920874797459 us; error 0 us; skew 500 ppm
I20260620 02:01:14.797698 30737 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:14 dist-test-slave-wg14 krb5kdc[30006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920874, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:14.799974 30737 init.cc:377] Logged in from keytab as kudu/127.20.1.187@KRBTEST.COM (short username kudu)
I20260620 02:01:14.804436 30737 webserver.cc:533] Webserver started at http://127.20.1.187:46101/ using document root <none> and password file <none>
I20260620 02:01:14.804873 30737 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:14.804972 30737 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:14.809171 30737 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:01:14.812779 30866 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:14.813372 30737 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 02:01:14.813589 30737 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "05111c362ece43ed8f2e37fccbab0594"
format_stamp: "Formatted at 2026-06-20 02:01:14 on dist-test-slave-wg14"
I20260620 02:01:14.813781 30737 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:14.831086 30737 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:14.837512 30737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:14.935312 30737 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:37653
I20260620 02:01:14.935333 30917 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:37653 every 8 connection(s)
I20260620 02:01:14.936462 30737 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 02:01:14.946767 30918 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:01:14.949635 30918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: Bootstrap starting.
I20260620 02:01:14.994007 30918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:01:14.995117 30918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: Bootstrap complete.
I20260620 02:01:14.997712 30918 raft_consensus.cc:359] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: true } }
I20260620 02:01:14.998626 30918 raft_consensus.cc:740] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05111c362ece43ed8f2e37fccbab0594, State: Initialized, Role: LEARNER
I20260620 02:01:14.999922 30918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: true } }
I20260620 02:01:15.011793 30918 sys_catalog.cc:565] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:01:15.013701 30922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: true } } }
I20260620 02:01:15.014211 30922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:01:15.029752 30893 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260620 02:01:15.030886 30922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fff0de072014bd498aa189ff340dea7. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: true } } }
I20260620 02:01:15.031230 30922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:01:15.048482 30933 catalog_manager.cc:1294] Loaded cluster ID: 0354708363be4f35a33f4b05654c434f
I20260620 02:01:15.048669 30933 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: loading cluster ID for follower catalog manager: success
I20260620 02:01:15.053709 30933 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: acquiring CA information for follower catalog manager: success
I20260620 02:01:15.056510 30933 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:01:15.063870 30826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: true } } }
I20260620 02:01:15.064409 30826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: This master's current role is: LEADER
W20260620 02:01:15.333413 30853 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:01:15.333824 30853 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:01:15.333990 30853 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:01:15.347085 30853 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:01:15.347478 30853 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:01:15.361362 30853 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:38373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=43585
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391,127.20.1.187:37653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:01:15.365446 30853 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:01:15.368099 30853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:15.386588 30939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:15.387322 30938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:15.388839 30941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:15.389974 30853 server_base.cc:1061] running on GCE node
I20260620 02:01:15.391261 30853 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:01:15.391534 30853 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:01:15.391659 30853 hybrid_clock.cc:648] HybridClock initialized: now 1781920875391630 us; error 0 us; skew 500 ppm
I20260620 02:01:15.392140 30853 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:15 dist-test-slave-wg14 krb5kdc[30006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920875, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:15.397315 30853 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:01:15.399986 30853 webserver.cc:533] Webserver started at http://127.20.1.129:43585/ using document root <none> and password file <none>
I20260620 02:01:15.401098 30853 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:15.401496 30853 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:15.412039 30853 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:01:15.418432 30947 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:15.421615 30853 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.007s sys 0.000s
I20260620 02:01:15.422433 30853 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "0fdcf25fbea94ad8a67f10a023127832"
format_stamp: "Formatted at 2026-06-20 02:01:12 on dist-test-slave-wg14"
I20260620 02:01:15.423661 30853 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:15.428835 30845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: true } } }
I20260620 02:01:15.429286 30845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:15.431847 30826 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7: attempting to promote NON_VOTER 05111c362ece43ed8f2e37fccbab0594 to VOTER
I20260620 02:01:15.433091 30826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } }
I20260620 02:01:15.436808 30422 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [term 1 FOLLOWER]: Refusing update from remote peer 4fff0de072014bd498aa189ff340dea7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:01:15.436789 30893 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [term 1 LEARNER]: Refusing update from remote peer 4fff0de072014bd498aa189ff340dea7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:01:15.437117 30348 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [term 1 FOLLOWER]: Refusing update from remote peer 4fff0de072014bd498aa189ff340dea7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:01:15.437872 30845 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:01:15.438398 30826 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:01:15.438738 30844 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:01:15.444963 30845 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 05111c362ece43ed8f2e37fccbab0594 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } } }
I20260620 02:01:15.446462 30348 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 05111c362ece43ed8f2e37fccbab0594 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } } }
I20260620 02:01:15.451874 30853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:15.451570 30844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } } }
I20260620 02:01:15.451570 30826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } } }
I20260620 02:01:15.452247 30844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:15.452565 30844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } } }
I20260620 02:01:15.451617 30422 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 05111c362ece43ed8f2e37fccbab0594 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } } }
I20260620 02:01:15.452837 30844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:15.452278 30826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:15.459415 30955 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } } }
I20260620 02:01:15.459669 30853 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:01:15.460060 30853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:15.461233 30955 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:15.462713 30954 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } } }
I20260620 02:01:15.463022 30954 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2ce249d2ef54e3c80bb6ca0c2e310f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:15.466257 30853 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:01:15.466365 30845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } } }
I20260620 02:01:15.466930 30845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:15.468319 30893 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 05111c362ece43ed8f2e37fccbab0594 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } } }
I20260620 02:01:15.471179 30853 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:01:15.471357 30853 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.002s
I20260620 02:01:15.471475 30853 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:01:15.471555 30853 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:15.478724 30922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } } }
I20260620 02:01:15.479218 30922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05111c362ece43ed8f2e37fccbab0594 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:15.537950 30853 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38373
I20260620 02:01:15.538036 31063 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38373 every 8 connection(s)
I20260620 02:01:15.541719 30853 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:01:15.548668 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 30853
Jun 20 02:01:15 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920875, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
Jun 20 02:01:15 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920875, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 02:01:15 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920875, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:01:15.607532 31064 heartbeater.cc:344] Connected to a master server at 127.20.1.187:37653
I20260620 02:01:15.608067 31064 heartbeater.cc:461] Registering TS with master...
I20260620 02:01:15.611210 31064 heartbeater.cc:507] Master 127.20.1.187:37653 requested a full tablet report, sending...
I20260620 02:01:15.615721 30883 ts_manager.cc:194] Registered new tserver with Master: 0fdcf25fbea94ad8a67f10a023127832 (127.20.1.129:38373)
I20260620 02:01:15.619398 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Jun 20 02:01:15 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920875, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:01:15.621924 31065 heartbeater.cc:344] Connected to a master server at 127.20.1.188:43391
I20260620 02:01:15.622156 31065 heartbeater.cc:461] Registering TS with master...
I20260620 02:01:15.623616 31066 heartbeater.cc:344] Connected to a master server at 127.20.1.190:42199
I20260620 02:01:15.623696 31065 heartbeater.cc:507] Master 127.20.1.188:43391 requested a full tablet report, sending...
I20260620 02:01:15.624059 31066 heartbeater.cc:461] Registering TS with master...
I20260620 02:01:15.625331 31066 heartbeater.cc:507] Master 127.20.1.190:42199 requested a full tablet report, sending...
I20260620 02:01:15.627550 30491 ts_manager.cc:194] Re-registered known tserver with Master: 0fdcf25fbea94ad8a67f10a023127832 (127.20.1.129:38373)
I20260620 02:01:15.630599 30337 ts_manager.cc:194] Re-registered known tserver with Master: 0fdcf25fbea94ad8a67f10a023127832 (127.20.1.129:38373)
I20260620 02:01:15.630749 30491 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:51513
I20260620 02:01:15.637240 31067 heartbeater.cc:344] Connected to a master server at 127.20.1.189:42723
I20260620 02:01:15.637501 31067 heartbeater.cc:461] Registering TS with master...
I20260620 02:01:15.637960 31067 heartbeater.cc:507] Master 127.20.1.189:42723 requested a full tablet report, sending...
I20260620 02:01:15.639678 30413 ts_manager.cc:194] Re-registered known tserver with Master: 0fdcf25fbea94ad8a67f10a023127832 (127.20.1.129:38373)
I20260620 02:01:15.640669 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 02:01:15.640857 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 30307
I20260620 02:01:15.643390 30368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:01:15.794241 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 30307
I20260620 02:01:15.819638 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42199
--webserver_interface=127.20.1.190
--webserver_port=33757
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391
--master_addresses=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391,127.20.1.187:37653 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:01:15.990477 30478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.190:42199: connect: Connection refused (error 111)
W20260620 02:01:15.993217 30478 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 -> Peer f3b6016fc44444139ac8878b81be5835 (127.20.1.190:42199): Couldn't send request to peer f3b6016fc44444139ac8878b81be5835. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:42199: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260620 02:01:16.380990 31080 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:01:16.381912 31080 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:01:16.382050 31080 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:01:16.396791 31080 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:01:16.396939 31080 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:01:16.396999 31080 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:01:16.397060 31080 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:01:16.405565 30953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } } }
I20260620 02:01:16.406054 30953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:16.416287 31080 master_runner.cc:387] Master server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391,127.20.1.187:37653
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=33757
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:01:16.418382 31080 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:01:16.420282 31080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:16.434777 31090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:16.435590 31091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:16.435828 31093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:16.438377 31080 server_base.cc:1061] running on GCE node
I20260620 02:01:16.439687 31080 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:01:16.439920 31080 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:01:16.440037 31080 hybrid_clock.cc:648] HybridClock initialized: now 1781920876440005 us; error 0 us; skew 500 ppm
I20260620 02:01:16.440434 31080 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:16 dist-test-slave-wg14 krb5kdc[30006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920876, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:16.446341 31080 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:01:16.449436 31080 webserver.cc:533] Webserver started at http://127.20.1.190:33757/ using document root <none> and password file <none>
I20260620 02:01:16.450565 31080 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:16.450738 31080 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:16.462431 31080 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.006s sys 0.003s
I20260620 02:01:16.467309 31099 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:16.469951 31080 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260620 02:01:16.470342 31080 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "f3b6016fc44444139ac8878b81be5835"
format_stamp: "Formatted at 2026-06-20 02:01:09 on dist-test-slave-wg14"
I20260620 02:01:16.472851 31080 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:16.504416 31080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:16.511035 31080 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:01:16.511969 31080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:16.536520 30459 master_runner.cc:187] Error getting master registration for 127.20.1.190:42199: Network error: Client connection negotiation failed: client connection to 127.20.1.190:42199: connect: Connection refused (error 111),
I20260620 02:01:16.536823 30459 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:01:16.545823 31080 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:42199
I20260620 02:01:16.545991 31150 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:42199 every 8 connection(s)
I20260620 02:01:16.548159 31080 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:01:16.552645 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 31080
I20260620 02:01:16.553009 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 30853
I20260620 02:01:16.562053 31151 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:01:16.568981 31151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835: Bootstrap starting.
I20260620 02:01:16.590040 31059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:01:16.617015 31151 log.cc:826] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835: Log is configured to *not* fsync() on all Append() calls
I20260620 02:01:16.633096 31151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:01:16.634969 31151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835: Bootstrap complete.
I20260620 02:01:16.650192 31151 raft_consensus.cc:359] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } }
I20260620 02:01:16.653003 31151 raft_consensus.cc:740] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b6016fc44444139ac8878b81be5835, State: Initialized, Role: FOLLOWER
I20260620 02:01:16.656404 31151 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } }
I20260620 02:01:16.658026 31157 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } } }
I20260620 02:01:16.658743 31157 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:16.661019 31151 sys_catalog.cc:565] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:01:16.729848 31168 catalog_manager.cc:1294] Loaded cluster ID: 0354708363be4f35a33f4b05654c434f
I20260620 02:01:16.730126 31168 catalog_manager.cc:1587] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835: loading cluster ID for follower catalog manager: success
I20260620 02:01:16.746430 31168 catalog_manager.cc:1609] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835: acquiring CA information for follower catalog manager: success
I20260620 02:01:16.762516 31168 catalog_manager.cc:1637] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Jun 20 02:01:16 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920876, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:01:16.807235 31157 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fff0de072014bd498aa189ff340dea7. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } } }
I20260620 02:01:16.807538 31157 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b6016fc44444139ac8878b81be5835 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:16.832501 31065 heartbeater.cc:499] Master 127.20.1.188:43391 was elected leader, sending a full tablet report...
I20260620 02:01:16.840649 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 30853
W20260620 02:01:16.876155 31171 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:01:16.837576 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:52083 (local address 127.20.1.190:42199)
0620 02:01:16.838410 (+ 834us) server_negotiation.cc:207] Beginning negotiation
0620 02:01:16.838417 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 02:01:16.874175 (+ 35758us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:52083: BlockingRecv error: recv got EOF from 127.20.1.129:52083 (error 108)
Metrics: {"server-negotiator.queue_time_us":603}
W20260620 02:01:16.877127 30475 connection.cc:573] server connection from 127.20.1.129:51513 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:01:16.879616 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:38373
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391
--webserver_port=43585
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391,127.20.1.187:37653 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:01:17.350975 31087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fff0de072014bd498aa189ff340dea7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f3b6016fc44444139ac8878b81be5835" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42199 } } peers { permanent_uuid: "f2ce249d2ef54e3c80bb6ca0c2e310f2" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 42723 } } peers { permanent_uuid: "4fff0de072014bd498aa189ff340dea7" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 43391 } } peers { permanent_uuid: "05111c362ece43ed8f2e37fccbab0594" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37653 } attrs { promote: false } } }
I20260620 02:01:17.352738 31087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fff0de072014bd498aa189ff340dea7 [sys.catalog]: This master's current role is: LEADER
W20260620 02:01:17.440325 31178 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:01:17.440701 31178 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:01:17.440852 31178 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:01:17.452682 31178 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:01:17.452929 31178 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:01:17.467417 31178 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:38373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=43585
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391,127.20.1.187:37653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:01:17.470376 31178 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:01:17.473285 31178 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:17.492831 31187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:17.495072 31190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:17.495427 31188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:17.502056 31178 server_base.cc:1061] running on GCE node
I20260620 02:01:17.503643 31178 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:01:17.503965 31178 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:01:17.504083 31178 hybrid_clock.cc:648] HybridClock initialized: now 1781920877504054 us; error 0 us; skew 500 ppm
I20260620 02:01:17.504514 31178 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:17 dist-test-slave-wg14 krb5kdc[30006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:17.509859 31178 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:01:17.514151 31178 webserver.cc:533] Webserver started at http://127.20.1.129:43585/ using document root <none> and password file <none>
I20260620 02:01:17.515269 31178 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:17.515427 31178 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:17.524215 31178 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:01:17.529491 31196 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:17.533301 31178 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260620 02:01:17.533581 31178 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "0fdcf25fbea94ad8a67f10a023127832"
format_stamp: "Formatted at 2026-06-20 02:01:12 on dist-test-slave-wg14"
I20260620 02:01:17.534765 31178 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:17.574545 31178 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:17.583956 31178 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:01:17.584722 31178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:17.586616 31178 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:01:17.589453 31178 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:01:17.589614 31178 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:17.589751 31178 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:01:17.589840 31178 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:17.643100 31178 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38373
I20260620 02:01:17.643180 31308 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38373 every 8 connection(s)
I20260620 02:01:17.645630 31178 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:01:17.656140 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 31178
Jun 20 02:01:17 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Jun 20 02:01:17 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 02:01:17 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:01:17.718739 31310 heartbeater.cc:344] Connected to a master server at 127.20.1.188:43391
I20260620 02:01:17.719391 31310 heartbeater.cc:461] Registering TS with master...
I20260620 02:01:17.719996 31311 heartbeater.cc:344] Connected to a master server at 127.20.1.190:42199
I20260620 02:01:17.720142 31311 heartbeater.cc:461] Registering TS with master...
I20260620 02:01:17.722378 31310 heartbeater.cc:507] Master 127.20.1.188:43391 requested a full tablet report, sending...
I20260620 02:01:17.722592 31311 heartbeater.cc:507] Master 127.20.1.190:42199 requested a full tablet report, sending...
Jun 20 02:01:17 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:01:17.726102 30491 ts_manager.cc:194] Re-registered known tserver with Master: 0fdcf25fbea94ad8a67f10a023127832 (127.20.1.129:38373)
I20260620 02:01:17.727065 31116 ts_manager.cc:194] Registered new tserver with Master: 0fdcf25fbea94ad8a67f10a023127832 (127.20.1.129:38373)
I20260620 02:01:17.727456 30491 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:52899
Jun 20 02:01:17 dist-test-slave-wg14 krb5kdc[30006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
I20260620 02:01:17.742144 31309 heartbeater.cc:344] Connected to a master server at 127.20.1.187:37653
I20260620 02:01:17.742376 31309 heartbeater.cc:461] Registering TS with master...
I20260620 02:01:17.742812 31309 heartbeater.cc:507] Master 127.20.1.187:37653 requested a full tablet report, sending...
I20260620 02:01:17.745069 30882 ts_manager.cc:194] Re-registered known tserver with Master: 0fdcf25fbea94ad8a67f10a023127832 (127.20.1.129:38373)
I20260620 02:01:17.748236 31312 heartbeater.cc:344] Connected to a master server at 127.20.1.189:42723
I20260620 02:01:17.748481 31312 heartbeater.cc:461] Registering TS with master...
I20260620 02:01:17.748822 31312 heartbeater.cc:507] Master 127.20.1.189:42723 requested a full tablet report, sending...
I20260620 02:01:17.750856 30413 ts_manager.cc:194] Re-registered known tserver with Master: 0fdcf25fbea94ad8a67f10a023127832 (127.20.1.129:38373)
I20260620 02:01:17.904202 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 31178
W20260620 02:01:17.905368 31327 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:01:17.900730 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:44808 (local address 127.20.1.129:38373)
0620 02:01:17.902011 (+ 1281us) server_negotiation.cc:207] Beginning negotiation
0620 02:01:17.902020 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0620 02:01:17.902055 (+ 35us) server_negotiation.cc:408] Connection header received
0620 02:01:17.902132 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:01:17.902140 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:01:17.902394 (+ 254us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:01:17.902624 (+ 230us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:01:17.903985 (+ 1361us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44808: BlockingRecv error: recv got EOF from 127.0.0.1:44808 (error 108)
Metrics: {"server-negotiator.queue_time_us":1127,"thread_start_us":965,"threads_started":1}
W20260620 02:01:17.907189 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 31178: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:38373: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:01:17.907363 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 31178
I20260620 02:01:17.925127 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:38373
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391
--webserver_port=43585
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391,127.20.1.187:37653
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:01:18.418421 31329 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:01:18.418807 31329 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:01:18.418942 31329 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 02:01:18.418995 31329 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:01:18.433216 31329 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:01:18.433482 31329 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:01:18.448899 31329 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:38373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=43585
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391,127.20.1.187:37653
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:01:18.450902 31329 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:01:18.453156 31329 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:18.469590 31334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:18.469841 31335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:18.473016 31337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:18.473230 31329 server_base.cc:1061] running on GCE node
I20260620 02:01:18.474176 31329 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:01:18.474443 31329 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:01:18.474597 31329 hybrid_clock.cc:648] HybridClock initialized: now 1781920878474543 us; error 0 us; skew 500 ppm
I20260620 02:01:18.475283 31329 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:18 dist-test-slave-wg14 krb5kdc[30006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920878, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:18.481850 31329 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:01:18.485740 31329 webserver.cc:533] Webserver started at http://127.20.1.129:43585/ using document root <none> and password file <none>
I20260620 02:01:18.486879 31329 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:18.487036 31329 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:18.498821 31329 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.000s sys 0.004s
I20260620 02:01:18.503574 31343 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:18.507411 31329 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260620 02:01:18.507676 31329 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "0fdcf25fbea94ad8a67f10a023127832"
format_stamp: "Formatted at 2026-06-20 02:01:12 on dist-test-slave-wg14"
I20260620 02:01:18.508659 31329 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:18.553712 31329 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:18.562959 31329 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:01:18.563395 31329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:18.564788 31329 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:01:18.568102 31329 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:01:18.568246 31329 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:18.568410 31329 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:01:18.568472 31329 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 02:01:18.568989 31329 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 02:01:18.569073 31329 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 02:01:18.569166 31329 fault_injection.cc:59] injecting fault for kudu (pid 31329): FLAGS_fault_before_start (process will exit)
I20260620 02:01:18.592221 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:38373
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391
--webserver_port=43585
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391,127.20.1.187:37653 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:01:19.129079 31351 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:01:19.129449 31351 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:01:19.130548 31351 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:01:19.141836 31351 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:01:19.142614 31351 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:01:19.159763 31351 tablet_server_runner.cc:78] Tablet server non-default flags:
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:38373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=43585
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:42199,127.20.1.189:42723,127.20.1.188:43391,127.20.1.187:37653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:01:19.162393 31351 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:01:19.168195 31351 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:19.184926 31358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:19.186162 31357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:19.191710 31351 server_base.cc:1061] running on GCE node
W20260620 02:01:19.192517 31360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:19.193437 31351 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 02:01:19.193647 31351 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 02:01:19.193723 31351 hybrid_clock.cc:648] HybridClock initialized: now 1781920879193702 us; error 0 us; skew 500 ppm
I20260620 02:01:19.194088 31351 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:19 dist-test-slave-wg14 krb5kdc[30006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920879, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:19.199280 31351 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:01:19.213248 31351 webserver.cc:533] Webserver started at http://127.20.1.129:43585/ using document root <none> and password file <none>
I20260620 02:01:19.214819 31351 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:19.215219 31351 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:19.226514 31351 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.006s sys 0.000s
I20260620 02:01:19.233489 31366 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:19.240027 31351 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.005s sys 0.001s
I20260620 02:01:19.240309 31351 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "0fdcf25fbea94ad8a67f10a023127832"
format_stamp: "Formatted at 2026-06-20 02:01:12 on dist-test-slave-wg14"
I20260620 02:01:19.242044 31351 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:19.274266 31351 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:19.286586 31351 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:01:19.286985 31351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:19.289151 31351 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:01:19.293370 31351 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:01:19.293674 31351 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260620 02:01:19.294126 31351 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:01:19.294247 31351 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:19.374666 31351 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:38373
I20260620 02:01:19.374956 31478 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:38373 every 8 connection(s)
I20260620 02:01:19.376952 31351 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:01:19.385190 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 31351
I20260620 02:01:19.385564 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 31351
W20260620 02:01:19.407848 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 31351: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:38373: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:01:19.408150 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 31351
I20260620 02:01:19.428920 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 31080
W20260620 02:01:19.429077 31491 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:01:19.405198 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:34227 (local address 127.20.1.190:42199)
0620 02:01:19.419963 (+ 14765us) server_negotiation.cc:207] Beginning negotiation
0620 02:01:19.419969 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0620 02:01:19.428627 (+ 8658us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:34227: BlockingRecv error: recv got EOF from 127.20.1.129:34227 (error 108)
Metrics: {"server-negotiator.queue_time_us":14627,"thread_start_us":14364,"threads_started":1}
W20260620 02:01:19.429844 31484 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:01:19.384847 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:56681 (local address 127.20.1.188:43391)
0620 02:01:19.395430 (+ 10583us) server_negotiation.cc:207] Beginning negotiation
0620 02:01:19.395440 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0620 02:01:19.400638 (+ 5198us) server_negotiation.cc:408] Connection header received
0620 02:01:19.405497 (+ 4859us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:01:19.405505 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:01:19.405595 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:01:19.405712 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:01:19.428785 (+ 23073us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:56681: BlockingRecv error: recv got EOF from 127.20.1.129:56681 (error 108)
Metrics: {"server-negotiator.queue_time_us":10373,"thread_start_us":10538,"threads_started":1}
W20260620 02:01:19.429847 31488 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:01:19.403507 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:59481 (local address 127.20.1.187:37653)
0620 02:01:19.404060 (+ 553us) server_negotiation.cc:207] Beginning negotiation
0620 02:01:19.404067 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 02:01:19.407240 (+ 3173us) server_negotiation.cc:408] Connection header received
0620 02:01:19.428693 (+ 21453us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:59481: BlockingRecv error: recv got EOF from 127.20.1.129:59481 (error 108)
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":274,"threads_started":1}
W20260620 02:01:19.432178 31491 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:01:19.430050 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:42632 (local address 127.20.1.190:42199)
0620 02:01:19.430345 (+ 295us) server_negotiation.cc:207] Beginning negotiation
0620 02:01:19.430351 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0620 02:01:19.430587 (+ 236us) server_negotiation.cc:408] Connection header received
0620 02:01:19.430799 (+ 212us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:01:19.430806 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:01:19.430891 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:01:19.430992 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:01:19.431969 (+ 977us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42632: BlockingRecv error: recv got EOF from 127.0.0.1:42632 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
W20260620 02:01:19.432484 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 31080: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:42199: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:01:19.432684 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 31080
I20260620 02:01:19.456758 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 30382
I20260620 02:01:19.458606 30443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:01:19.612761 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 30382
I20260620 02:01:19.646219 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 30459
I20260620 02:01:19.648495 30521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:01:19.850803 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 30459
I20260620 02:01:19.896279 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 30737
I20260620 02:01:19.898705 30913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:01:20.104117 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 30737
Shutting down hive metastore.
[ OK ] ExternalMiniClusterTest.TestAddMaster/9 (43232 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/10
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:01:20 dist-test-slave-wg14 krb5kdc[31502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:01:20 dist-test-slave-wg14 krb5kdc[31502](info): set up 2 sockets
Jun 20 02:01:20 dist-test-slave-wg14 krb5kdc[31502](info): commencing operation
krb5kdc: starting...
W20260620 02:01:23.096113 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.488s user 0.004s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 02:01:23 dist-test-slave-wg14 krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:01:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:01:23Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5.conf
2026-06-20 02:01:37: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Jun 20 02:01:40 dist-test-slave-wg14 krb5kdc[31502](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920900, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 02:01:51 dist-test-slave-wg14 krb5kdc[31502](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920911, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 02:01:51 dist-test-slave-wg14 krb5kdc[31502](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920911, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:01:52.792201 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40321
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:01:53.271462 31806 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:01:53.272423 31806 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:01:53.272742 31806 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:01:53.285948 31806 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:01:53.286134 31806 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:01:53.286219 31806 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:01:53.286505 31806 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:01:53.304176 31806 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:01:53.306923 31806 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:01:53.309653 31806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:53.325377 31812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:53.325462 31811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:53.328995 31814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:53.329401 31806 server_base.cc:1061] running on GCE node
I20260620 02:01:53.331233 31806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:01:53.333920 31806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:01:53.337873 31806 hybrid_clock.cc:648] HybridClock initialized: now 1781920913337532 us; error 319 us; skew 500 ppm
I20260620 02:01:53.338502 31806 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:53 dist-test-slave-wg14 krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920913, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:53.343405 31806 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:01:53.346243 31806 webserver.cc:533] Webserver started at http://127.20.1.190:41145/ using document root <none> and password file <none>
I20260620 02:01:53.347386 31806 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:53.347704 31806 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:53.348551 31806 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:01:53.352599 31806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "bc85a1f42d124ecc8da866273ccfcf7c"
format_stamp: "Formatted at 2026-06-20 02:01:53 on dist-test-slave-wg14"
I20260620 02:01:53.353713 31806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "bc85a1f42d124ecc8da866273ccfcf7c"
format_stamp: "Formatted at 2026-06-20 02:01:53 on dist-test-slave-wg14"
I20260620 02:01:53.361857 31806 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260620 02:01:53.367476 31821 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:53.370249 31806 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 02:01:53.371244 31806 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "bc85a1f42d124ecc8da866273ccfcf7c"
format_stamp: "Formatted at 2026-06-20 02:01:53 on dist-test-slave-wg14"
I20260620 02:01:53.371790 31806 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:53.405215 31806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:53.417716 31806 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:01:53.418095 31806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:53.450501 31806 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:40321
I20260620 02:01:53.450525 31872 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:40321 every 8 connection(s)
I20260620 02:01:53.452463 31806 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:01:53.458161 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 31806
I20260620 02:01:53.458705 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:01:53.460083 31873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:01:53.471161 31873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:01:53 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920913, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.189@KRBTEST.COM" created.
I20260620 02:01:53.514927 31873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } has no permanent_uuid. Determining permanent_uuid...
W20260620 02:01:53.518347 31824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36161: connect: Connection refused (error 111)
W20260620 02:01:53.524008 31873 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36161: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36161: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:01:53.573695 31873 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } attempt: 1
W20260620 02:01:53.576208 31873 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36161: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36161: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:01:53.637204 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:36161
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:01:53.644630 31873 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } attempt: 2
W20260620 02:01:53.648310 31873 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36161: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36161: connect: Connection refused (error 111)
I20260620 02:01:53.739737 31873 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } attempt: 3
W20260620 02:01:53.744503 31873 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36161: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36161: connect: Connection refused (error 111)
I20260620 02:01:53.887874 31873 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } attempt: 4
W20260620 02:01:53.892721 31873 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36161: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36161: connect: Connection refused (error 111)
W20260620 02:01:54.154527 31882 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:01:54.154976 31882 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:01:54.155086 31882 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:01:54.166290 31882 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:01:54.166464 31882 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:01:54.166543 31882 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:01:54.166884 31882 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:01:54.192080 31873 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } attempt: 5
I20260620 02:01:54.194425 31882 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:36161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:01:54.201732 31882 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:01:54.204252 31882 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:54.204442 31873 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:36161: Network error: Client connection negotiation failed: client connection to 127.20.1.189:36161: connect: Connection refused (error 111)
W20260620 02:01:54.231448 31888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:54.232249 31887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:54.233633 31890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:54.235351 31882 server_base.cc:1061] running on GCE node
I20260620 02:01:54.244189 31882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:01:54.246755 31882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:01:54.248366 31882 hybrid_clock.cc:648] HybridClock initialized: now 1781920914248282 us; error 80 us; skew 500 ppm
I20260620 02:01:54.248934 31882 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:54 dist-test-slave-wg14 krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920914, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:54.260663 31882 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 02:01:54.268220 31882 webserver.cc:533] Webserver started at http://127.20.1.189:36095/ using document root <none> and password file <none>
I20260620 02:01:54.270458 31882 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:54.271399 31882 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:54.272372 31882 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:01:54.277139 31882 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "eb689aaa27774adfb3b859df4f358172"
format_stamp: "Formatted at 2026-06-20 02:01:54 on dist-test-slave-wg14"
I20260620 02:01:54.278278 31882 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "eb689aaa27774adfb3b859df4f358172"
format_stamp: "Formatted at 2026-06-20 02:01:54 on dist-test-slave-wg14"
I20260620 02:01:54.293354 31882 fs_manager.cc:696] Time spent creating directory manager: real 0.015s user 0.015s sys 0.001s
I20260620 02:01:54.306931 31897 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:54.309127 31882 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260620 02:01:54.309387 31882 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "eb689aaa27774adfb3b859df4f358172"
format_stamp: "Formatted at 2026-06-20 02:01:54 on dist-test-slave-wg14"
I20260620 02:01:54.309633 31882 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:54.353960 31882 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:54.362637 31882 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:01:54.362988 31882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:54.409982 31882 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:36161
I20260620 02:01:54.409983 31948 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:36161 every 8 connection(s)
I20260620 02:01:54.412211 31882 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 02:01:54.420643 31949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:01:54.421528 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 31882
I20260620 02:01:54.421855 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 02:01:54.433049 31949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:01:54 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920914, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
I20260620 02:01:54.469592 31949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.188@KRBTEST.COM" created.
Jun 20 02:01:54 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920914, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:01:54.521554 31949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } has no permanent_uuid. Determining permanent_uuid...
W20260620 02:01:54.524655 31901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46713: connect: Connection refused (error 111)
W20260620 02:01:54.527112 31949 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46713: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46713: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:01:54.540802 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.188: real 0.118s user 0.004s sys 0.036s
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.188@KRBTEST.COM" created.
I20260620 02:01:54.576661 31949 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } attempt: 1
W20260620 02:01:54.580503 31949 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46713: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46713: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:01:54.627274 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:46713
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:01:54.648890 31949 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } attempt: 2
W20260620 02:01:54.651667 31949 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46713: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46713: connect: Connection refused (error 111)
I20260620 02:01:54.742969 31949 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } attempt: 3
W20260620 02:01:54.746344 31949 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46713: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46713: connect: Connection refused (error 111)
I20260620 02:01:54.751906 31873 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } attempt: 6
Jun 20 02:01:54 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920913, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:01:54.786159 31873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } has no permanent_uuid. Determining permanent_uuid...
W20260620 02:01:54.789258 31873 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46713: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46713: connect: Connection refused (error 111)
I20260620 02:01:54.841624 31873 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } attempt: 1
W20260620 02:01:54.845187 31873 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46713: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46713: connect: Connection refused (error 111)
I20260620 02:01:54.889763 31949 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } attempt: 4
W20260620 02:01:54.892467 31949 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46713: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46713: connect: Connection refused (error 111)
I20260620 02:01:54.919584 31873 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } attempt: 2
W20260620 02:01:54.923542 31873 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46713: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46713: connect: Connection refused (error 111)
I20260620 02:01:55.037357 31873 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } attempt: 3
W20260620 02:01:55.041635 31873 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46713: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46713: connect: Connection refused (error 111)
I20260620 02:01:55.191027 31873 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } attempt: 4
I20260620 02:01:55.191851 31949 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } attempt: 5
W20260620 02:01:55.194213 31949 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46713: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46713: connect: Connection refused (error 111)
W20260620 02:01:55.194352 31873 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46713: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46713: connect: Connection refused (error 111)
W20260620 02:01:55.216611 31959 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:01:55.217098 31959 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:01:55.217257 31959 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:01:55.227900 31959 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:01:55.228055 31959 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:01:55.228149 31959 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:01:55.228212 31959 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:01:55.240947 31959 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:46713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:01:55.243418 31959 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:01:55.246430 31959 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:55.262461 31969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:55.262421 31967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:55.262652 31966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:55.264243 31959 server_base.cc:1061] running on GCE node
I20260620 02:01:55.265547 31959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:01:55.268893 31959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:01:55.272707 31959 hybrid_clock.cc:648] HybridClock initialized: now 1781920915272574 us; error 105 us; skew 500 ppm
I20260620 02:01:55.273587 31959 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:55 dist-test-slave-wg14 krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920915, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:55.279101 31959 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 02:01:55.282317 31959 webserver.cc:533] Webserver started at http://127.20.1.188:43225/ using document root <none> and password file <none>
I20260620 02:01:55.283438 31959 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:55.283605 31959 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:55.283980 31959 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:01:55.288810 31959 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "02fb7c1e8c8e49068920fc2b8ce7273f"
format_stamp: "Formatted at 2026-06-20 02:01:55 on dist-test-slave-wg14"
I20260620 02:01:55.290802 31959 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "02fb7c1e8c8e49068920fc2b8ce7273f"
format_stamp: "Formatted at 2026-06-20 02:01:55 on dist-test-slave-wg14"
I20260620 02:01:55.302066 31959 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.006s sys 0.001s
I20260620 02:01:55.308880 31976 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:55.312031 31959 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.006s sys 0.001s
I20260620 02:01:55.312497 31959 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "02fb7c1e8c8e49068920fc2b8ce7273f"
format_stamp: "Formatted at 2026-06-20 02:01:55 on dist-test-slave-wg14"
I20260620 02:01:55.312779 31959 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:55.347002 31959 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:55.355353 31959 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:01:55.355729 31959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:55.395434 31959 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:46713
I20260620 02:01:55.395444 32027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:46713 every 8 connection(s)
I20260620 02:01:55.397508 31959 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 02:01:55.401177 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 31959
I20260620 02:01:55.401908 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 02:01:55.408999 32028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:01:55.436167 32028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:01:55 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 02:01:55 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920915, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:01:55.463047 31873 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } attempt: 5
Jun 20 02:01:55 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:01:55.481813 32028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:01:55 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920913, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 02:01:55 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920915, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 02:01:55 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:01:55.522444 31873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c: Bootstrap starting.
I20260620 02:01:55.528626 31873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c: Neither blocks nor log segments found. Creating new log.
I20260620 02:01:55.529330 32028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:01:55.530812 31873 log.cc:826] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:01:55.535327 31873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c: No bootstrap required, opened a new log
Jun 20 02:01:55 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920915, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:01:55.542788 31873 raft_consensus.cc:359] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } }
I20260620 02:01:55.543213 31873 raft_consensus.cc:385] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:01:55.543390 31873 raft_consensus.cc:740] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc85a1f42d124ecc8da866273ccfcf7c, State: Initialized, Role: FOLLOWER
I20260620 02:01:55.544206 31873 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } }
I20260620 02:01:55.548164 32039 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } }
I20260620 02:01:55.548650 32039 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:55.548909 32039 raft_consensus.cc:493] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:01:55.549167 32039 raft_consensus.cc:515] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } }
I20260620 02:01:55.552661 32039 leader_election.cc:290] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb689aaa27774adfb3b859df4f358172 (127.20.1.189:36161), 02fb7c1e8c8e49068920fc2b8ce7273f (127.20.1.188:46713)
I20260620 02:01:55.555395 31873 sys_catalog.cc:565] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:01:55.564635 32028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f: Bootstrap starting.
I20260620 02:01:55.572242 32028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f: Neither blocks nor log segments found. Creating new log.
I20260620 02:01:55.574680 32028 log.cc:826] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:01:55.580686 32028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f: No bootstrap required, opened a new log
I20260620 02:01:55.596765 32028 raft_consensus.cc:359] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } }
I20260620 02:01:55.597368 32028 raft_consensus.cc:385] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:01:55.597532 32028 raft_consensus.cc:740] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb7c1e8c8e49068920fc2b8ce7273f, State: Initialized, Role: FOLLOWER
I20260620 02:01:55.598682 32028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } }
I20260620 02:01:55.600582 32052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } }
I20260620 02:01:55.601379 32052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:55.601851 32028 sys_catalog.cc:565] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:01:55.602689 32003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" is_pre_election: true
I20260620 02:01:55.604215 32003 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc85a1f42d124ecc8da866273ccfcf7c in term 0.
I20260620 02:01:55.605690 31825 leader_election.cc:304] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb7c1e8c8e49068920fc2b8ce7273f, bc85a1f42d124ecc8da866273ccfcf7c; no voters:
I20260620 02:01:55.606400 32039 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:01:55.606642 32039 raft_consensus.cc:493] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:01:55.606774 32039 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 0 FOLLOWER]: Advancing to term 1
W20260620 02:01:55.609534 31982 tablet.cc:2406] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:01:55.611954 32039 raft_consensus.cc:515] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } }
I20260620 02:01:55.617100 32003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f"
I20260620 02:01:55.618127 32003 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 0 FOLLOWER]: Advancing to term 1
W20260620 02:01:55.618660 32051 catalog_manager.cc:1593] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:01:55.618862 32051 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:01:55.621618 32003 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc85a1f42d124ecc8da866273ccfcf7c in term 1.
I20260620 02:01:55.622062 32039 leader_election.cc:290] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [CANDIDATE]: Term 1 election: Requested vote from peers eb689aaa27774adfb3b859df4f358172 (127.20.1.189:36161), 02fb7c1e8c8e49068920fc2b8ce7273f (127.20.1.188:46713)
I20260620 02:01:55.622381 31825 leader_election.cc:304] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb7c1e8c8e49068920fc2b8ce7273f, bc85a1f42d124ecc8da866273ccfcf7c; no voters:
I20260620 02:01:55.623206 32039 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:01:55.625031 32039 raft_consensus.cc:697] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 1 LEADER]: Becoming Leader. State: Replica: bc85a1f42d124ecc8da866273ccfcf7c, State: Running, Role: LEADER
I20260620 02:01:55.605788 31806 master_runner.cc:187] Error getting master registration for 127.20.1.189:36161: OK, instance_id { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" instance_seqno: 1781920914398547 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:01:55.629410 31806 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:01:55.635501 32039 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } }
I20260620 02:01:55.653743 32038 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc85a1f42d124ecc8da866273ccfcf7c. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } }
I20260620 02:01:55.654211 32038 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:55.656641 32067 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
W20260620 02:01:55.659849 32064 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:01:55.660135 32064 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:01:55.662685 32067 catalog_manager.cc:1519] Initializing Kudu cluster ID...
W20260620 02:01:55.674290 31827 tablet.cc:2406] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:01:55.681493 32003 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 1 FOLLOWER]: Refusing update from remote peer bc85a1f42d124ecc8da866273ccfcf7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:01:55.670279 31959 master_runner.cc:187] Error getting master registration for 127.20.1.189:36161: OK, instance_id { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" instance_seqno: 1781920914398547 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:01:55.684834 31959 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:01:55.684684 32038 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:01:55.720933 32052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc85a1f42d124ecc8da866273ccfcf7c. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } }
I20260620 02:01:55.721356 32052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:55.733714 32069 mvcc.cc:204] Tried to move back new op lower bound from 7298748070614994944 to 7298748070453321728. Current Snapshot: MvccSnapshot[applied={T|T < 7298748070614994944}]
I20260620 02:01:55.737071 32052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } }
I20260620 02:01:55.737455 32052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:55.741482 31949 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } attempt: 6
I20260620 02:01:55.750831 32039 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } }
I20260620 02:01:55.751300 32039 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:55.752562 32067 catalog_manager.cc:1382] Generated new cluster ID: 5646b342100f424086a33dabcf8e71c7
I20260620 02:01:55.752696 32067 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
Jun 20 02:01:55 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920914, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:01:55.791790 32067 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:01:55.794351 32067 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:01:55.812971 31949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172: Bootstrap starting.
I20260620 02:01:55.818496 31949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172: Neither blocks nor log segments found. Creating new log.
I20260620 02:01:55.826545 32067 catalog_manager.cc:6069] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c: Generated new TSK 0
I20260620 02:01:55.828576 31949 log.cc:826] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172: Log is configured to *not* fsync() on all Append() calls
I20260620 02:01:55.829214 32067 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 02:01:55.849581 31949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172: No bootstrap required, opened a new log
I20260620 02:01:55.867470 31949 raft_consensus.cc:359] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } }
I20260620 02:01:55.868691 31949 raft_consensus.cc:385] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:01:55.869107 31949 raft_consensus.cc:740] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb689aaa27774adfb3b859df4f358172, State: Initialized, Role: FOLLOWER
I20260620 02:01:55.870923 31949 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } }
I20260620 02:01:55.873613 32076 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } }
I20260620 02:01:55.875602 32076 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:55.877760 31923 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb689aaa27774adfb3b859df4f358172"
I20260620 02:01:55.878304 31922 raft_consensus.cc:3060] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:01:55.881434 31949 sys_catalog.cc:565] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:01:55.877774 31924 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb689aaa27774adfb3b859df4f358172" is_pre_election: true
I20260620 02:01:55.887588 31922 raft_consensus.cc:1275] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [term 1 FOLLOWER]: Refusing update from remote peer bc85a1f42d124ecc8da866273ccfcf7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:01:55.889058 32039 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
I20260620 02:01:55.939041 32076 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc85a1f42d124ecc8da866273ccfcf7c. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } }
I20260620 02:01:55.939545 32076 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:55.939919 32083 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } }
I20260620 02:01:55.940639 32083 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:55.993975 32093 catalog_manager.cc:1294] Loaded cluster ID: 5646b342100f424086a33dabcf8e71c7
I20260620 02:01:55.994236 32093 catalog_manager.cc:1587] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172: loading cluster ID for follower catalog manager: success
I20260620 02:01:55.999018 32093 catalog_manager.cc:1609] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172: acquiring CA information for follower catalog manager: success
I20260620 02:01:56.002947 32093 catalog_manager.cc:1637] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:01:56.054054 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.207s user 0.005s sys 0.034s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:01:56.209359 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.155s user 0.003s sys 0.031s
I20260620 02:01:56.210860 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:01:56.294023 32039 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } }
I20260620 02:01:56.294368 32039 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:56.663719 32064 catalog_manager.cc:1294] Loaded cluster ID: 5646b342100f424086a33dabcf8e71c7
I20260620 02:01:56.663940 32064 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f: loading cluster ID for follower catalog manager: success
I20260620 02:01:56.669536 32064 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f: acquiring CA information for follower catalog manager: success
I20260620 02:01:56.676285 32064 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 02:01:56.847268 32098 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:01:56.847857 32098 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:01:56.848444 32098 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:01:56.859700 32098 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:01:56.859926 32098 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:01:56.873221 32098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:01:56.875425 32098 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:01:56.877249 32098 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:56.891996 32103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:56.892688 32106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:56.891959 32104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:56.894270 32098 server_base.cc:1061] running on GCE node
I20260620 02:01:56.895428 32098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:01:56.896848 32098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:01:56.898268 32098 hybrid_clock.cc:648] HybridClock initialized: now 1781920916898167 us; error 46 us; skew 500 ppm
I20260620 02:01:56.898942 32098 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:56 dist-test-slave-wg14 krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920916, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:56.903606 32098 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:01:56.906108 32098 webserver.cc:533] Webserver started at http://127.20.1.129:40323/ using document root <none> and password file <none>
I20260620 02:01:56.907068 32098 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:56.907260 32098 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:56.907653 32098 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:01:56.910758 32098 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "f6447fdb0c814671b8bea8396458b02a"
format_stamp: "Formatted at 2026-06-20 02:01:56 on dist-test-slave-wg14"
I20260620 02:01:56.911917 32098 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "f6447fdb0c814671b8bea8396458b02a"
format_stamp: "Formatted at 2026-06-20 02:01:56 on dist-test-slave-wg14"
I20260620 02:01:56.921504 32098 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.003s sys 0.005s
I20260620 02:01:56.927711 32113 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:56.930783 32098 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.002s
I20260620 02:01:56.931090 32098 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "f6447fdb0c814671b8bea8396458b02a"
format_stamp: "Formatted at 2026-06-20 02:01:56 on dist-test-slave-wg14"
I20260620 02:01:56.931896 32098 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:56.995335 32098 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:57.000453 32098 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:01:57.000783 32098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:57.001928 32098 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:01:57.004312 32098 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:01:57.004432 32098 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:57.004581 32098 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:01:57.004637 32098 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:57.060333 32098 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:44591
I20260620 02:01:57.060386 32225 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:44591 every 8 connection(s)
I20260620 02:01:57.062230 32098 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:01:57.068519 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 32098
I20260620 02:01:57.068763 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 02:01:57 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920916, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 02:01:57 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920916, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 02:01:57 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920916, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:01:57.113591 32227 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40321
I20260620 02:01:57.114116 32227 heartbeater.cc:461] Registering TS with master...
I20260620 02:01:57.114794 32228 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36161
I20260620 02:01:57.114998 32228 heartbeater.cc:461] Registering TS with master...
I20260620 02:01:57.115875 32227 heartbeater.cc:507] Master 127.20.1.190:40321 requested a full tablet report, sending...
I20260620 02:01:57.115931 32228 heartbeater.cc:507] Master 127.20.1.189:36161 requested a full tablet report, sending...
I20260620 02:01:57.119234 31837 ts_manager.cc:194] Registered new tserver with Master: f6447fdb0c814671b8bea8396458b02a (127.20.1.129:44591)
I20260620 02:01:57.119242 31914 ts_manager.cc:194] Registered new tserver with Master: f6447fdb0c814671b8bea8396458b02a (127.20.1.129:44591)
I20260620 02:01:57.119316 32226 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46713
I20260620 02:01:57.119477 32226 heartbeater.cc:461] Registering TS with master...
I20260620 02:01:57.120265 32226 heartbeater.cc:507] Master 127.20.1.188:46713 requested a full tablet report, sending...
I20260620 02:01:57.122383 31837 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:44327
I20260620 02:01:57.122987 31993 ts_manager.cc:194] Registered new tserver with Master: f6447fdb0c814671b8bea8396458b02a (127.20.1.129:44591)
I20260620 02:01:57.131718 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
WARNING: no policy specified for kudu/127.20.1.187@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.187@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.187 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.187 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.187@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.187@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.187 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.187 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:01:57.318195 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:42075
--webserver_interface=127.20.1.187
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.187
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713,127.20.1.187:42075 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:01:57.923559 32240 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:01:57.924050 32240 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:01:57.924226 32240 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:01:57.947376 32240 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:01:57.948004 32240 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:01:57.948325 32240 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:01:57.948413 32240 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:01:57.971709 32240 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713,127.20.1.187:42075
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.187
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:42075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.20.1.187
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:01:57.975677 32240 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:01:57.977569 32240 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:57.991287 32245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:57.991667 32246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:57.993305 32248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:57.998329 32240 server_base.cc:1061] running on GCE node
I20260620 02:01:58.005764 32240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:01:58.008126 32240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:01:58.009539 32240 hybrid_clock.cc:648] HybridClock initialized: now 1781920918009452 us; error 77 us; skew 500 ppm
I20260620 02:01:58.010262 32240 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:58 dist-test-slave-wg14 krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920918, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:58.017267 32240 init.cc:377] Logged in from keytab as kudu/127.20.1.187@KRBTEST.COM (short username kudu)
I20260620 02:01:58.026832 32240 webserver.cc:533] Webserver started at http://127.20.1.187:37961/ using document root <none> and password file <none>
I20260620 02:01:58.027915 32240 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:58.028090 32240 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:58.028556 32240 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:01:58.033726 32240 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/data/instance:
uuid: "5df7bdff65dd4a4884588076d137f173"
format_stamp: "Formatted at 2026-06-20 02:01:58 on dist-test-slave-wg14"
I20260620 02:01:58.036726 32240 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/wal/instance:
uuid: "5df7bdff65dd4a4884588076d137f173"
format_stamp: "Formatted at 2026-06-20 02:01:58 on dist-test-slave-wg14"
I20260620 02:01:58.051043 32240 fs_manager.cc:696] Time spent creating directory manager: real 0.014s user 0.010s sys 0.002s
I20260620 02:01:58.058197 32255 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:58.066352 32240 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.007s sys 0.001s
I20260620 02:01:58.066648 32240 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "5df7bdff65dd4a4884588076d137f173"
format_stamp: "Formatted at 2026-06-20 02:01:58 on dist-test-slave-wg14"
I20260620 02:01:58.066908 32240 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:58.113529 32240 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:58.119266 32240 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:01:58.124814 32240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:58.129072 32227 heartbeater.cc:499] Master 127.20.1.190:40321 was elected leader, sending a full tablet report...
I20260620 02:01:58.173285 32240 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:42075
I20260620 02:01:58.173285 32306 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:42075 every 8 connection(s)
I20260620 02:01:58.176539 32240 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 02:01:58.185618 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 32240
I20260620 02:01:58.189706 32307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:01:58.207741 32307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:01:58 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
Jun 20 02:01:58 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920918, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:01:58.249239 32307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:01:58 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920918, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:01:58.275918 32307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:01:58 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920918, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:01:58.301965 32307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:01:58 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920918, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
I20260620 02:01:58.330600 32307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: Bootstrap starting.
I20260620 02:01:58.336352 32307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: Neither blocks nor log segments found. Creating new log.
I20260620 02:01:58.337779 32307 log.cc:826] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: Log is configured to *not* fsync() on all Append() calls
I20260620 02:01:58.341547 32307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: No bootstrap required, opened a new log
I20260620 02:01:58.350600 32307 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } }
I20260620 02:01:58.351322 32307 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:01:58.351933 32307 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df7bdff65dd4a4884588076d137f173, State: Initialized, Role: FOLLOWER
I20260620 02:01:58.353155 32307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } }
I20260620 02:01:58.357952 32307 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:01:58.359027 32316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } } }
I20260620 02:01:58.360477 32316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 02:01:58.373140 32327 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:01:58.373332 32327 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 02:01:58.378898 32261 tablet.cc:2406] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:01:58.394711 32240 master_runner.cc:429] Detected that this master 5df7bdff65dd4a4884588076d137f173 is joining an existing cluster
I20260620 02:01:58.394929 32240 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260620 02:01:58.408171 32240 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 02:01:58.408967 32240 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 02:01:58.409540 32240 tablet_replica.cc:333] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: stopping tablet replica
I20260620 02:01:58.414238 32240 master_runner.cc:434] Initiating AddMaster RPC to add 127.20.1.187:42075
Jun 20 02:01:58 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920913, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
I20260620 02:01:58.574666 31837 catalog_manager.cc:7201] Initiating ChangeConfig request to add master 127.20.1.187:42075: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5df7bdff65dd4a4884588076d137f173"
member_type: NON_VOTER
last_known_addr {
host: "127.20.1.187"
port: 42075
}
attrs {
promote: true
}
}
dest_uuid: "bc85a1f42d124ecc8da866273ccfcf7c"
cas_config_opid_index: -1
I20260620 02:01:58.577539 31837 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: true } }
W20260620 02:01:58.587219 31825 consensus_peers.cc:597] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c -> Peer 5df7bdff65dd4a4884588076d137f173 (127.20.1.187:42075): Couldn't send request to peer 5df7bdff65dd4a4884588076d137f173. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260620 02:01:58.587334 31922 raft_consensus.cc:1275] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [term 1 FOLLOWER]: Refusing update from remote peer bc85a1f42d124ecc8da866273ccfcf7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 02:01:58.587607 32002 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 1 FOLLOWER]: Refusing update from remote peer bc85a1f42d124ecc8da866273ccfcf7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 02:01:58.588433 32340 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 02:01:58.588793 32038 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 02:01:58.593403 32038 raft_consensus.cc:2955] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5df7bdff65dd4a4884588076d137f173 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: true } } }
I20260620 02:01:58.601928 31922 raft_consensus.cc:2955] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5df7bdff65dd4a4884588076d137f173 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: true } } }
I20260620 02:01:58.607301 32038 catalog_manager.cc:7205] Successfully completed master ChangeConfig request to add master 127.20.1.187:42075
I20260620 02:01:58.608336 32240 master.cc:562] Master@127.20.1.187:42075 shutting down...
I20260620 02:01:58.613142 32340 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: true } } }
I20260620 02:01:58.613461 32340 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:58.617861 32340 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: true } } }
I20260620 02:01:58.618260 32340 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:58.611721 32344 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: true } } }
I20260620 02:01:58.621816 32344 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:58.625468 32340 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: true } } }
I20260620 02:01:58.625818 32340 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:58.627928 32003 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5df7bdff65dd4a4884588076d137f173 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: true } } }
I20260620 02:01:58.634914 32240 master.cc:584] Master@127.20.1.187:42075 shutdown complete.
I20260620 02:01:58.635100 32240 master_runner.cc:306] Clearing existing system tablet
I20260620 02:01:58.636318 32240 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:58.636629 32240 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:58.647904 32346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: true } } }
I20260620 02:01:58.648502 32346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:58.652145 32240 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.004s sys 0.004s
I20260620 02:01:58.654330 32348 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:58.657934 32240 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:58.660245 32240 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260620 02:01:58.660403 32240 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "5df7bdff65dd4a4884588076d137f173"
format_stamp: "Formatted at 2026-06-20 02:01:58 on dist-test-slave-wg14"
I20260620 02:01:58.665452 32240 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:01:58.671842 32240 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 02:01:58.672093 32240 log.cc:1199] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: Deleting WAL directory at /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260620 02:01:58.672731 32240 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: Deleting consensus metadata
I20260620 02:01:58.673815 32240 master_runner.cc:316] Copying system tablet from 127.20.1.190:40321
I20260620 02:01:58.681519 32240 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: tablet copy: Beginning tablet copy session from remote peer at address 127.20.1.190:40321
I20260620 02:01:58.706977 31858 tablet_copy_service.cc:140] P bc85a1f42d124ecc8da866273ccfcf7c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5df7bdff65dd4a4884588076d137f173 ({username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:45480)
I20260620 02:01:58.708093 31858 tablet_copy_service.cc:161] P bc85a1f42d124ecc8da866273ccfcf7c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5df7bdff65dd4a4884588076d137f173 at {username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:45480: session id = 5df7bdff65dd4a4884588076d137f173-00000000000000000000000000000000
I20260620 02:01:58.716070 31858 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 02:01:58.722306 32240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 02:01:58.728469 32240 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: tablet copy: Starting download of 0 data blocks...
I20260620 02:01:58.728929 32240 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: tablet copy: Starting download of 1 WAL segments...
I20260620 02:01:58.733853 32240 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 02:01:58.741325 31858 tablet_copy_service.cc:342] P bc85a1f42d124ecc8da866273ccfcf7c: Request end of tablet copy session 5df7bdff65dd4a4884588076d137f173-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:45480
I20260620 02:01:58.741653 31858 tablet_copy_service.cc:434] P bc85a1f42d124ecc8da866273ccfcf7c: ending tablet copy session 5df7bdff65dd4a4884588076d137f173-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5df7bdff65dd4a4884588076d137f173
W20260620 02:01:58.750658 32240 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260620 02:01:58.756918 32240 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:58.764115 32356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:58.764947 32355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:58.771546 32240 server_base.cc:1061] running on GCE node
W20260620 02:01:58.771564 32359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:58.772178 32240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:01:58.772881 32240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:01:58.774151 32240 hybrid_clock.cc:648] HybridClock initialized: now 1781920918774072 us; error 69 us; skew 500 ppm
I20260620 02:01:58.774466 32240 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:58 dist-test-slave-wg14 krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920918, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:58.787235 32240 init.cc:377] Logged in from keytab as kudu/127.20.1.187@KRBTEST.COM (short username kudu)
I20260620 02:01:58.790463 32240 webserver.cc:533] Webserver started at http://127.20.1.187:45099/ using document root <none> and password file <none>
I20260620 02:01:58.790973 32240 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:58.791158 32240 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:58.795161 32240 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 02:01:58.801990 32365 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:58.804768 32240 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260620 02:01:58.805044 32240 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "5df7bdff65dd4a4884588076d137f173"
format_stamp: "Formatted at 2026-06-20 02:01:58 on dist-test-slave-wg14"
I20260620 02:01:58.805274 32240 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:58.840734 32240 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:58.848762 32240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:58.921330 32240 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:42075
I20260620 02:01:58.921526 32416 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:42075 every 8 connection(s)
I20260620 02:01:58.923794 32240 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 02:01:58.928982 32417 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:01:58.932534 32417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: Bootstrap starting.
I20260620 02:01:58.971997 32417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:01:58.972873 32417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: Bootstrap complete.
I20260620 02:01:58.975852 32417 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: true } }
I20260620 02:01:58.978668 32417 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5df7bdff65dd4a4884588076d137f173, State: Initialized, Role: LEARNER
I20260620 02:01:58.983000 32417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: true } }
I20260620 02:01:58.985236 32417 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:01:58.985136 32422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: true } } }
I20260620 02:01:58.985633 32422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:01:59.014418 32433 catalog_manager.cc:1294] Loaded cluster ID: 5646b342100f424086a33dabcf8e71c7
I20260620 02:01:59.014586 32433 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: loading cluster ID for follower catalog manager: success
I20260620 02:01:59.022444 32392 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260620 02:01:59.026340 32422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc85a1f42d124ecc8da866273ccfcf7c. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: true } } }
I20260620 02:01:59.026778 32422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:01:59.031986 32433 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: acquiring CA information for follower catalog manager: success
I20260620 02:01:59.038383 32433 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:01:59.042621 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 32098
Jun 20 02:01:59 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:01:59.075831 32221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:01:59.268045 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 32098
I20260620 02:01:59.298092 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:44591
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713
--webserver_port=40323
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713,127.20.1.187:42075 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:01:59.496366 32343 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: true } } }
I20260620 02:01:59.496809 32343 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:59.499782 32438 raft_consensus.cc:1064] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c: attempting to promote NON_VOTER 5df7bdff65dd4a4884588076d137f173 to VOTER
I20260620 02:01:59.501418 32438 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } }
I20260620 02:01:59.504429 32003 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 1 FOLLOWER]: Refusing update from remote peer bc85a1f42d124ecc8da866273ccfcf7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:01:59.505089 32392 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 1 LEARNER]: Refusing update from remote peer bc85a1f42d124ecc8da866273ccfcf7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:01:59.505117 31922 raft_consensus.cc:1275] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [term 1 FOLLOWER]: Refusing update from remote peer bc85a1f42d124ecc8da866273ccfcf7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:01:59.506167 32343 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:01:59.507261 32343 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:01:59.508042 32343 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:01:59.517220 32437 raft_consensus.cc:2955] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5df7bdff65dd4a4884588076d137f173 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:01:59.518028 31922 raft_consensus.cc:2955] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5df7bdff65dd4a4884588076d137f173 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:01:59.518345 32392 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5df7bdff65dd4a4884588076d137f173 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:01:59.523300 32422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:01:59.523669 32422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:59.524734 32438 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:01:59.525034 32438 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:59.525144 32003 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5df7bdff65dd4a4884588076d137f173 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:01:59.525473 32438 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:01:59.525879 32438 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:59.526360 32438 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:01:59.526584 32438 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:59.526885 32437 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:01:59.527088 32437 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: This master's current role is: LEADER
I20260620 02:01:59.530151 32442 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:01:59.530571 32442 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:01:59.535002 32443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:01:59.535372 32443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: This master's current role is: FOLLOWER
W20260620 02:01:59.828867 32436 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:01:59.829267 32436 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:01:59.829406 32436 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:01:59.841405 32436 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:01:59.841853 32436 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:01:59.855183 32436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:44591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=40323
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713,127.20.1.187:42075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:01:59.857548 32436 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:01:59.859431 32436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:01:59.875638 32448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:59.875970 32449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:01:59.878182 32451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:01:59.878782 32436 server_base.cc:1061] running on GCE node
I20260620 02:01:59.879802 32436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:01:59.882841 32436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:01:59.884366 32436 hybrid_clock.cc:648] HybridClock initialized: now 1781920919884237 us; error 103 us; skew 500 ppm
I20260620 02:01:59.885615 32436 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:01:59 dist-test-slave-wg14 krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920919, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:01:59.889945 32436 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:01:59.892663 32436 webserver.cc:533] Webserver started at http://127.20.1.129:40323/ using document root <none> and password file <none>
I20260620 02:01:59.894073 32436 fs_manager.cc:362] Metadata directory not provided
I20260620 02:01:59.894307 32436 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:01:59.903223 32436 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:01:59.909094 32458 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:59.912528 32436 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260620 02:01:59.912818 32436 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "f6447fdb0c814671b8bea8396458b02a"
format_stamp: "Formatted at 2026-06-20 02:01:56 on dist-test-slave-wg14"
I20260620 02:01:59.913858 32436 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:01:59.942601 32436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:01:59.948556 32436 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:01:59.948984 32436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:01:59.950320 32436 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:01:59.952435 32436 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:01:59.952561 32436 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:01:59.952657 32436 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:01:59.952747 32436 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:00.005693 32436 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:44591
I20260620 02:02:00.005744 32570 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:44591 every 8 connection(s)
I20260620 02:02:00.009277 32436 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:02:00.017875 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 32436
Jun 20 02:02:00 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920919, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 02:02:00 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920919, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 02:02:00 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920919, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
Jun 20 02:02:00 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920919, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:02:00.073191 32574 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36161
I20260620 02:02:00.073993 32574 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:00.075675 32574 heartbeater.cc:507] Master 127.20.1.189:36161 requested a full tablet report, sending...
I20260620 02:02:00.077260 32573 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40321
I20260620 02:02:00.077301 32571 heartbeater.cc:344] Connected to a master server at 127.20.1.187:42075
I20260620 02:02:00.077512 32573 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:00.077517 32571 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:00.078320 32573 heartbeater.cc:507] Master 127.20.1.190:40321 requested a full tablet report, sending...
I20260620 02:02:00.078338 32571 heartbeater.cc:507] Master 127.20.1.187:42075 requested a full tablet report, sending...
I20260620 02:02:00.078557 31914 ts_manager.cc:194] Re-registered known tserver with Master: f6447fdb0c814671b8bea8396458b02a (127.20.1.129:44591)
I20260620 02:02:00.080963 32382 ts_manager.cc:194] Registered new tserver with Master: f6447fdb0c814671b8bea8396458b02a (127.20.1.129:44591)
I20260620 02:02:00.081027 31837 ts_manager.cc:194] Re-registered known tserver with Master: f6447fdb0c814671b8bea8396458b02a (127.20.1.129:44591)
I20260620 02:02:00.082610 31837 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:36957
I20260620 02:02:00.085584 32572 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46713
I20260620 02:02:00.085815 32572 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:00.086261 32572 heartbeater.cc:507] Master 127.20.1.188:46713 requested a full tablet report, sending...
I20260620 02:02:00.088804 31993 ts_manager.cc:194] Re-registered known tserver with Master: f6447fdb0c814671b8bea8396458b02a (127.20.1.129:44591)
I20260620 02:02:00.089253 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 02:02:00.111253 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 02:02:00.111609 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 31806
I20260620 02:02:00.113722 31868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:02:00.268746 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 31806
W20260620 02:02:00.292505 31898 connection.cc:573] server connection from 127.0.0.1:39886 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:02:00.293224 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40321
--webserver_interface=127.20.1.190
--webserver_port=41145
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713
--master_addresses=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713,127.20.1.187:42075 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:02:00.690652 31959 master_runner.cc:187] Error getting master registration for 127.20.1.190:40321: Network error: Client connection negotiation failed: client connection to 127.20.1.190:40321: connect: Connection refused (error 111),
I20260620 02:02:00.690981 31959 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 02:02:00.930364 32587 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:02:00.930862 32587 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:02:00.931032 32587 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:02:00.943728 32587 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:02:00.943931 32587 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:02:00.944017 32587 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:02:00.944181 32587 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:02:00.965183 32587 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713,127.20.1.187:42075
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=41145
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:02:00.967314 32587 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:02:00.969583 32587 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:02:00.991492 32594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:00.992084 32593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:00.993952 32596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:00.995380 32587 server_base.cc:1061] running on GCE node
I20260620 02:02:00.996451 32587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:02:01.005148 32587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:02:01.006567 32587 hybrid_clock.cc:648] HybridClock initialized: now 1781920921006497 us; error 70 us; skew 500 ppm
I20260620 02:02:01.007681 32587 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:02:01 dist-test-slave-wg14 krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920921, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:02:01.013180 32587 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:02:01.025894 32587 webserver.cc:533] Webserver started at http://127.20.1.190:41145/ using document root <none> and password file <none>
I20260620 02:02:01.027269 32587 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:01.027483 32587 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:01.046902 32587 fs_manager.cc:714] Time spent opening directory manager: real 0.017s user 0.016s sys 0.001s
I20260620 02:02:01.052327 32603 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:01.055751 32587 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:02:01.056020 32587 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "bc85a1f42d124ecc8da866273ccfcf7c"
format_stamp: "Formatted at 2026-06-20 02:01:53 on dist-test-slave-wg14"
I20260620 02:02:01.057029 32587 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:01.086452 32573 heartbeater.cc:499] Master 127.20.1.190:40321 was elected leader, sending a full tablet report...
W20260620 02:02:01.088944 32573 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:40321 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:40321: connect: Connection refused (error 111)
I20260620 02:02:01.088970 32587 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:02:01.100345 32587 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:02:01.100865 32587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:02:01.138006 32655 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:40321 every 8 connection(s)
I20260620 02:02:01.138131 32587 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:40321
I20260620 02:02:01.141536 32587 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:02:01.145486 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 32587
I20260620 02:02:01.145876 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 32436
I20260620 02:02:01.153438 32656 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:02:01.163609 32656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c: Bootstrap starting.
I20260620 02:02:01.174418 32566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:02:01.202733 32656 log.cc:826] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:02:01.229192 32656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:02:01.230115 32656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c: Bootstrap complete.
I20260620 02:02:01.244647 32656 raft_consensus.cc:359] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } }
I20260620 02:02:01.246225 32656 raft_consensus.cc:740] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc85a1f42d124ecc8da866273ccfcf7c, State: Initialized, Role: FOLLOWER
I20260620 02:02:01.247303 32656 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } }
I20260620 02:02:01.248782 32662 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:02:01.249301 32662 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:01.250761 32656 sys_catalog.cc:565] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:02:01.285758 32673 catalog_manager.cc:1294] Loaded cluster ID: 5646b342100f424086a33dabcf8e71c7
I20260620 02:02:01.285912 32673 catalog_manager.cc:1587] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c: loading cluster ID for follower catalog manager: success
I20260620 02:02:01.293457 32673 catalog_manager.cc:1609] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c: acquiring CA information for follower catalog manager: success
Jun 20 02:02:01 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920921, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:02:01.302031 32673 catalog_manager.cc:1637] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:02:01.341872 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 32436
I20260620 02:02:01.364363 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:44591
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713
--webserver_port=40323
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713,127.20.1.187:42075 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:02:01.401176 32587 master_runner.cc:264] No leader master found from master bc85a1f42d124ecc8da866273ccfcf7c
I20260620 02:02:01.401405 32587 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:02:01.612180 32681 raft_consensus.cc:493] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bc85a1f42d124ecc8da866273ccfcf7c)
I20260620 02:02:01.612778 32681 raft_consensus.cc:515] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } }
I20260620 02:02:01.615980 32682 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bc85a1f42d124ecc8da866273ccfcf7c)
I20260620 02:02:01.616379 32682 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } }
I20260620 02:02:01.621727 31922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "eb689aaa27774adfb3b859df4f358172" is_pre_election: true
I20260620 02:02:01.620770 32681 leader_election.cc:290] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc85a1f42d124ecc8da866273ccfcf7c (127.20.1.190:40321), eb689aaa27774adfb3b859df4f358172 (127.20.1.189:36161), 5df7bdff65dd4a4884588076d137f173 (127.20.1.187:42075)
I20260620 02:02:01.637655 32682 leader_election.cc:290] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc85a1f42d124ecc8da866273ccfcf7c (127.20.1.190:40321), eb689aaa27774adfb3b859df4f358172 (127.20.1.189:36161), 02fb7c1e8c8e49068920fc2b8ce7273f (127.20.1.188:46713)
I20260620 02:02:01.656958 32631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" is_pre_election: true
I20260620 02:02:01.658852 32631 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb7c1e8c8e49068920fc2b8ce7273f in term 1.
I20260620 02:02:01.662416 32392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5df7bdff65dd4a4884588076d137f173" is_pre_election: true
I20260620 02:02:01.662875 32392 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb7c1e8c8e49068920fc2b8ce7273f in term 1.
I20260620 02:02:01.664368 31980 leader_election.cc:304] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 02fb7c1e8c8e49068920fc2b8ce7273f, 5df7bdff65dd4a4884588076d137f173, bc85a1f42d124ecc8da866273ccfcf7c; no voters: eb689aaa27774adfb3b859df4f358172
I20260620 02:02:01.665119 32681 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 02:02:01.665257 32681 raft_consensus.cc:493] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 1 FOLLOWER]: Starting leader election (detected failure of leader bc85a1f42d124ecc8da866273ccfcf7c)
I20260620 02:02:01.665356 32681 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:02:01.670132 32681 raft_consensus.cc:515] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } }
I20260620 02:02:01.670631 32631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5df7bdff65dd4a4884588076d137f173" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" is_pre_election: true
I20260620 02:02:01.671043 32631 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5df7bdff65dd4a4884588076d137f173 in term 1.
I20260620 02:02:01.678159 31924 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5df7bdff65dd4a4884588076d137f173" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "eb689aaa27774adfb3b859df4f358172" is_pre_election: true
I20260620 02:02:01.678722 31922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "eb689aaa27774adfb3b859df4f358172"
I20260620 02:02:01.680275 32631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "bc85a1f42d124ecc8da866273ccfcf7c"
I20260620 02:02:01.681473 32631 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:02:01.682808 32003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5df7bdff65dd4a4884588076d137f173" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" is_pre_election: true
I20260620 02:02:01.683538 32003 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5df7bdff65dd4a4884588076d137f173 in current term 2: Already voted for candidate 02fb7c1e8c8e49068920fc2b8ce7273f in this term.
I20260620 02:02:01.684849 32369 leader_election.cc:304] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5df7bdff65dd4a4884588076d137f173, bc85a1f42d124ecc8da866273ccfcf7c; no voters: 02fb7c1e8c8e49068920fc2b8ce7273f, eb689aaa27774adfb3b859df4f358172
I20260620 02:02:01.686750 32682 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:02:01.687765 32681 leader_election.cc:290] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [CANDIDATE]: Term 2 election: Requested vote from peers bc85a1f42d124ecc8da866273ccfcf7c (127.20.1.190:40321), eb689aaa27774adfb3b859df4f358172 (127.20.1.189:36161), 5df7bdff65dd4a4884588076d137f173 (127.20.1.187:42075)
I20260620 02:02:01.689486 32392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5df7bdff65dd4a4884588076d137f173"
I20260620 02:02:01.692570 32682 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:02:01.697455 32392 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb7c1e8c8e49068920fc2b8ce7273f in term 2.
I20260620 02:02:01.699577 32631 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb7c1e8c8e49068920fc2b8ce7273f in term 2.
I20260620 02:02:01.700440 31980 leader_election.cc:304] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 02fb7c1e8c8e49068920fc2b8ce7273f, 5df7bdff65dd4a4884588076d137f173, bc85a1f42d124ecc8da866273ccfcf7c; no voters: eb689aaa27774adfb3b859df4f358172
I20260620 02:02:01.701215 32681 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:02:01.711383 32681 raft_consensus.cc:697] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [term 2 LEADER]: Becoming Leader. State: Replica: 02fb7c1e8c8e49068920fc2b8ce7273f, State: Running, Role: LEADER
I20260620 02:02:01.712586 32681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } }
I20260620 02:02:01.716805 32689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02fb7c1e8c8e49068920fc2b8ce7273f. Latest consensus state: current_term: 2 leader_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:02:01.717321 32689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: This master's current role is: LEADER
I20260620 02:02:01.718508 32691 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:02:01.722506 32691 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:02:01.723850 32691 catalog_manager.cc:1294] Loaded cluster ID: 5646b342100f424086a33dabcf8e71c7
I20260620 02:02:01.723989 32691 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:02:01.725811 32691 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:02:01.727162 32691 catalog_manager.cc:6080] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f: Loaded TSK: 0
I20260620 02:02:01.728459 32691 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 02:02:01.772631 32692 raft_consensus.cc:493] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bc85a1f42d124ecc8da866273ccfcf7c)
I20260620 02:02:01.773065 32692 raft_consensus.cc:515] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } }
I20260620 02:02:01.779248 32692 leader_election.cc:290] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc85a1f42d124ecc8da866273ccfcf7c (127.20.1.190:40321), 02fb7c1e8c8e49068920fc2b8ce7273f (127.20.1.188:46713), 5df7bdff65dd4a4884588076d137f173 (127.20.1.187:42075)
I20260620 02:02:01.781847 32003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb689aaa27774adfb3b859df4f358172" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" is_pre_election: true
I20260620 02:02:01.793232 32631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb689aaa27774adfb3b859df4f358172" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" is_pre_election: true
I20260620 02:02:01.793788 32631 raft_consensus.cc:2393] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb689aaa27774adfb3b859df4f358172 in current term 2: Already voted for candidate 02fb7c1e8c8e49068920fc2b8ce7273f in this term.
I20260620 02:02:01.795542 31901 leader_election.cc:304] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: eb689aaa27774adfb3b859df4f358172; no voters: 02fb7c1e8c8e49068920fc2b8ce7273f, bc85a1f42d124ecc8da866273ccfcf7c
I20260620 02:02:01.796103 32692 raft_consensus.cc:3060] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:02:01.799419 32692 raft_consensus.cc:2749] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:02:01.801126 32392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb689aaa27774adfb3b859df4f358172" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5df7bdff65dd4a4884588076d137f173" is_pre_election: true
I20260620 02:02:01.801504 32392 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb689aaa27774adfb3b859df4f358172 in current term 2: Already voted for candidate 02fb7c1e8c8e49068920fc2b8ce7273f in this term.
W20260620 02:02:01.981150 32679 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:02:01.981506 32679 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:02:01.981657 32679 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:02:01.994763 32679 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:02:01.995002 32679 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:02:02.011847 32679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:44591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=40323
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713,127.20.1.187:42075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:02:02.017249 32679 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:02:02.019038 32679 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:02:02.032449 32700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:02.032598 32699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:02.034518 32702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:02.035198 32679 server_base.cc:1061] running on GCE node
I20260620 02:02:02.035912 32679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:02:02.037081 32679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:02:02.038371 32679 hybrid_clock.cc:648] HybridClock initialized: now 1781920922038318 us; error 43 us; skew 500 ppm
I20260620 02:02:02.038777 32679 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:02:02 dist-test-slave-wg14 krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920922, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:02:02.043244 32679 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:02:02.045629 32679 webserver.cc:533] Webserver started at http://127.20.1.129:40323/ using document root <none> and password file <none>
I20260620 02:02:02.048673 32679 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:02.048877 32679 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:02.057950 32679 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 02:02:02.062661 32709 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:02.065619 32679 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 02:02:02.066704 32679 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "f6447fdb0c814671b8bea8396458b02a"
format_stamp: "Formatted at 2026-06-20 02:01:56 on dist-test-slave-wg14"
I20260620 02:02:02.068994 32679 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:02.089896 32679 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:02:02.097707 32679 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:02:02.098112 32679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:02:02.100164 32679 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:02:02.101181 32631 raft_consensus.cc:1275] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [term 2 FOLLOWER]: Refusing update from remote peer 02fb7c1e8c8e49068920fc2b8ce7273f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260620 02:02:02.102283 32689 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 02:02:02.103768 32679 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:02:02.103926 32679 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:02.104051 32679 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:02:02.104128 32679 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:02.108817 32716 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02fb7c1e8c8e49068920fc2b8ce7273f. Latest consensus state: current_term: 2 leader_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:02:02.109195 32716 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:02.130077 31922 raft_consensus.cc:1275] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [term 2 FOLLOWER]: Refusing update from remote peer 02fb7c1e8c8e49068920fc2b8ce7273f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260620 02:02:02.131702 32689 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 02:02:02.144160 32692 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02fb7c1e8c8e49068920fc2b8ce7273f. Latest consensus state: current_term: 2 leader_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:02:02.144603 32692 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:02.150214 32392 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [term 2 FOLLOWER]: Refusing update from remote peer 02fb7c1e8c8e49068920fc2b8ce7273f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260620 02:02:02.153061 32689 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 02:02:02.155900 32681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:02:02.157011 32681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: This master's current role is: LEADER
I20260620 02:02:02.157752 32689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:02:02.158013 32689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: This master's current role is: LEADER
I20260620 02:02:02.170964 32716 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:02:02.171286 32716 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc85a1f42d124ecc8da866273ccfcf7c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:02.171525 32692 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:02:02.171855 32692 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb689aaa27774adfb3b859df4f358172 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:02.176474 32682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02fb7c1e8c8e49068920fc2b8ce7273f. Latest consensus state: current_term: 2 leader_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:02:02.177205 32682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:02.176087 317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:02:02.177659 317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df7bdff65dd4a4884588076d137f173 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:02.198447 32679 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:44591
I20260620 02:02:02.198518 363 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:44591 every 8 connection(s)
I20260620 02:02:02.200556 32679 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:02:02.201133 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 32679
Jun 20 02:02:02 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Jun 20 02:02:02 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920922, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
Jun 20 02:02:02 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920922, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 02:02:02 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920922, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 02:02:02 dist-test-slave-wg14 krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920922, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:02:02.317999 371 heartbeater.cc:344] Connected to a master server at 127.20.1.189:36161
I20260620 02:02:02.318204 366 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40321
I20260620 02:02:02.318666 371 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:02.318677 366 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:02.321630 366 heartbeater.cc:507] Master 127.20.1.190:40321 requested a full tablet report, sending...
I20260620 02:02:02.322474 365 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46713
I20260620 02:02:02.322683 371 heartbeater.cc:507] Master 127.20.1.189:36161 requested a full tablet report, sending...
I20260620 02:02:02.322707 365 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:02.324074 365 heartbeater.cc:507] Master 127.20.1.188:46713 requested a full tablet report, sending...
I20260620 02:02:02.324939 31914 ts_manager.cc:194] Re-registered known tserver with Master: f6447fdb0c814671b8bea8396458b02a (127.20.1.129:44591)
I20260620 02:02:02.328752 32621 ts_manager.cc:194] Registered new tserver with Master: f6447fdb0c814671b8bea8396458b02a (127.20.1.129:44591)
I20260620 02:02:02.334465 31993 ts_manager.cc:194] Re-registered known tserver with Master: f6447fdb0c814671b8bea8396458b02a (127.20.1.129:44591)
I20260620 02:02:02.336942 31993 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:55355
I20260620 02:02:02.340039 364 heartbeater.cc:344] Connected to a master server at 127.20.1.187:42075
I20260620 02:02:02.340294 364 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:02.340879 364 heartbeater.cc:507] Master 127.20.1.187:42075 requested a full tablet report, sending...
I20260620 02:02:02.343171 32382 ts_manager.cc:194] Re-registered known tserver with Master: f6447fdb0c814671b8bea8396458b02a (127.20.1.129:44591)
I20260620 02:02:02.513688 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 32679
W20260620 02:02:02.516026 380 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:02:02.509610 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:50222 (local address 127.20.1.129:44591)
0620 02:02:02.510573 (+ 963us) server_negotiation.cc:207] Beginning negotiation
0620 02:02:02.510583 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0620 02:02:02.510621 (+ 38us) server_negotiation.cc:408] Connection header received
0620 02:02:02.510717 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:02:02.510725 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:02:02.511027 (+ 302us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:02:02.511305 (+ 278us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:02:02.513925 (+ 2620us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50222: BlockingRecv error: recv got EOF from 127.0.0.1:50222 (error 108)
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":581,"threads_started":1}
W20260620 02:02:02.517040 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 32679: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:44591: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:02:02.517339 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 32679
W20260620 02:02:02.517607 381 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:02:02.514768 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.20.1.129:44591)
0620 02:02:02.515649 (+ 881us) server_negotiation.cc:207] Beginning negotiation
0620 02:02:02.515655 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0620 02:02:02.515680 (+ 25us) server_negotiation.cc:408] Connection header received
0620 02:02:02.515745 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:02:02.515754 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:02:02.515825 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:02:02.515943 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:02:02.517275 (+ 1332us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50224: BlockingRecv error: recv got EOF from 127.0.0.1:50224 (error 108)
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":458,"threads_started":1}
I20260620 02:02:02.531893 32689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc85a1f42d124ecc8da866273ccfcf7c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40321 } } peers { permanent_uuid: "eb689aaa27774adfb3b859df4f358172" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 36161 } } peers { permanent_uuid: "02fb7c1e8c8e49068920fc2b8ce7273f" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46713 } } peers { permanent_uuid: "5df7bdff65dd4a4884588076d137f173" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 42075 } attrs { promote: false } } }
I20260620 02:02:02.532305 32689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f [sys.catalog]: This master's current role is: LEADER
I20260620 02:02:02.544958 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:44591
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713
--webserver_port=40323
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713,127.20.1.187:42075
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:02:03.194708 382 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:02:03.195104 382 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:02:03.195282 382 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 02:02:03.195360 382 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:02:03.213877 382 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:02:03.214216 382 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:02:03.233318 382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:44591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=40323
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713,127.20.1.187:42075
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:02:03.237998 382 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:02:03.241202 382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:02:03.269424 387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:03.269650 390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:03.269629 388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:03.271523 382 server_base.cc:1061] running on GCE node
I20260620 02:02:03.272744 382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:02:03.273979 382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:02:03.275404 382 hybrid_clock.cc:648] HybridClock initialized: now 1781920923275343 us; error 52 us; skew 500 ppm
I20260620 02:02:03.275827 382 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:02:03 dist-test-slave-wg14 krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920923, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:02:03.280309 382 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:02:03.283249 382 webserver.cc:533] Webserver started at http://127.20.1.129:40323/ using document root <none> and password file <none>
I20260620 02:02:03.284129 382 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:03.284276 382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:03.296150 382 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:02:03.300772 398 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:03.303026 382 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:02:03.303406 382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "f6447fdb0c814671b8bea8396458b02a"
format_stamp: "Formatted at 2026-06-20 02:01:56 on dist-test-slave-wg14"
I20260620 02:02:03.305363 382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:03.346757 382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:02:03.356540 382 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:02:03.356860 382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:02:03.358301 382 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:02:03.360520 382 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:02:03.360635 382 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:03.360749 382 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:02:03.360786 382 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 02:02:03.361446 382 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 02:02:03.361579 382 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 02:02:03.361666 382 fault_injection.cc:59] injecting fault for kudu (pid 382): FLAGS_fault_before_start (process will exit)
I20260620 02:02:03.386122 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:44591
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713
--webserver_port=40323
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713,127.20.1.187:42075 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:02:04.027117 406 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:02:04.027521 406 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:02:04.027657 406 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:02:04.048358 406 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:02:04.048902 406 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:02:04.073637 406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:44591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=40323
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:40321,127.20.1.189:36161,127.20.1.188:46713,127.20.1.187:42075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:02:04.079664 406 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:02:04.081475 406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:02:04.099788 414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:04.102545 412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:04.102680 411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:04.103412 406 server_base.cc:1061] running on GCE node
I20260620 02:02:04.104564 406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:02:04.106146 406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:02:04.107473 406 hybrid_clock.cc:648] HybridClock initialized: now 1781920924107412 us; error 61 us; skew 500 ppm
I20260620 02:02:04.107914 406 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:02:04 dist-test-slave-wg14 krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920924, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:02:04.112236 406 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:02:04.114709 406 webserver.cc:533] Webserver started at http://127.20.1.129:40323/ using document root <none> and password file <none>
I20260620 02:02:04.120159 406 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:04.120958 406 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:04.129964 406 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:02:04.134135 421 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:04.139521 406 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.005s sys 0.002s
I20260620 02:02:04.139811 406 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "f6447fdb0c814671b8bea8396458b02a"
format_stamp: "Formatted at 2026-06-20 02:01:56 on dist-test-slave-wg14"
I20260620 02:02:04.141669 406 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:04.191048 406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:02:04.207803 406 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:02:04.208088 406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:02:04.209515 406 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:02:04.211954 406 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:02:04.212096 406 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:04.212219 406 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:02:04.212363 406 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 02:02:04.309384 406 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:44591
I20260620 02:02:04.309442 534 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:44591 every 8 connection(s)
I20260620 02:02:04.311941 406 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:02:04.316221 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 406
I20260620 02:02:04.316579 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 406
W20260620 02:02:04.351069 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 406: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:44591: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:02:04.351305 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 406
W20260620 02:02:04.390341 539 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:02:04.317270 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:48101 (local address 127.20.1.188:46713)
0620 02:02:04.318100 (+ 830us) server_negotiation.cc:207] Beginning negotiation
0620 02:02:04.318108 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 02:02:04.340868 (+ 22760us) server_negotiation.cc:408] Connection header received
0620 02:02:04.347510 (+ 6642us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:02:04.347518 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:02:04.347648 (+ 130us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:02:04.347790 (+ 142us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:02:04.389151 (+ 41361us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:48101: BlockingRecv error: recv got EOF from 127.20.1.129:48101 (error 108)
Metrics: {"server-negotiator.queue_time_us":668,"spinlock_wait_cycles":13056,"thread_start_us":643,"threads_started":1}
W20260620 02:02:04.390337 547 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:02:04.344171 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:44279 (local address 127.20.1.190:40321)
0620 02:02:04.345907 (+ 1736us) server_negotiation.cc:207] Beginning negotiation
0620 02:02:04.345916 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0620 02:02:04.348707 (+ 2791us) server_negotiation.cc:408] Connection header received
0620 02:02:04.348996 (+ 289us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:02:04.349002 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:02:04.349085 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:02:04.349197 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:02:04.389214 (+ 40017us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:44279: BlockingRecv error: recv got EOF from 127.20.1.129:44279 (error 108)
Metrics: {"server-negotiator.queue_time_us":1553,"thread_start_us":1290,"threads_started":1}
W20260620 02:02:04.391070 549 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:02:04.347372 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:35267 (local address 127.20.1.187:42075)
0620 02:02:04.362810 (+ 15438us) server_negotiation.cc:207] Beginning negotiation
0620 02:02:04.362818 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 02:02:04.389317 (+ 26499us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:35267: BlockingRecv error: recv got EOF from 127.20.1.129:35267 (error 108)
Metrics: {"server-negotiator.queue_time_us":14418,"thread_start_us":15876,"threads_started":1}
I20260620 02:02:04.391616 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 32587
W20260620 02:02:04.392395 544 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:02:04.334016 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:59055 (local address 127.20.1.189:36161)
0620 02:02:04.334598 (+ 582us) server_negotiation.cc:207] Beginning negotiation
0620 02:02:04.334613 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0620 02:02:04.335242 (+ 629us) server_negotiation.cc:408] Connection header received
0620 02:02:04.347656 (+ 12414us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:02:04.347663 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:02:04.347754 (+ 91us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:02:04.347867 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:02:04.351054 (+ 3187us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 02:02:04.352914 (+ 1860us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:02:04.391189 (+ 38275us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:59055: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":284,"threads_started":1}
W20260620 02:02:04.405877 547 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:02:04.393887 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:40358 (local address 127.20.1.190:40321)
0620 02:02:04.394631 (+ 744us) server_negotiation.cc:207] Beginning negotiation
0620 02:02:04.394637 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0620 02:02:04.394666 (+ 29us) server_negotiation.cc:408] Connection header received
0620 02:02:04.394839 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:02:04.394844 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:02:04.394927 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:02:04.395038 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:02:04.405619 (+ 10581us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40358: BlockingRecv error: recv got EOF from 127.0.0.1:40358 (error 108)
Metrics: {"server-negotiator.queue_time_us":275}
W20260620 02:02:04.406232 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 32587: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:40321: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:02:04.406442 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 32587
I20260620 02:02:04.432974 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 31882
I20260620 02:02:04.435431 31944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:02:04.601418 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 31882
W20260620 02:02:04.622529 31980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.190:40321: connect: Connection refused (error 111)
W20260620 02:02:04.624815 31980 consensus_peers.cc:597] T 00000000000000000000000000000000 P 02fb7c1e8c8e49068920fc2b8ce7273f -> Peer bc85a1f42d124ecc8da866273ccfcf7c (127.20.1.190:40321): Couldn't send request to peer bc85a1f42d124ecc8da866273ccfcf7c. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:40321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 02:02:04.630856 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 31959
I20260620 02:02:04.632608 32023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:02:04.785208 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 31959
I20260620 02:02:04.818059 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 32240
I20260620 02:02:04.819379 32412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:02:04.981437 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 32240
Shutting down hive metastore.
2026-06-20T02:02:05Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/10 (44850 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/11
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:02:05 dist-test-slave-wg14 krb5kdc[561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:02:05 dist-test-slave-wg14 krb5kdc[561](info): set up 2 sockets
Jun 20 02:02:05 dist-test-slave-wg14 krb5kdc[561](info): commencing operation
krb5kdc: starting...
W20260620 02:02:08.147368 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.685s user 0.003s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 02:02:08 dist-test-slave-wg14 krb5kdc[561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:02:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:02:08Z Disabled control of system clock
2026-06-20T02:02:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:02:08Z Disabled control of system clock
2026-06-20T02:02:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:02:08Z Disabled control of system clock
2026-06-20T02:02:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:02:08Z Disabled control of system clock
2026-06-20T02:02:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:02:08Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5.conf
2026-06-20 02:02:22: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpZv5rS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Jun 20 02:02:26 dist-test-slave-wg14 krb5kdc[561](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920946, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 02:02:37 dist-test-slave-wg14 krb5kdc[561](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920957, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 02:02:37 dist-test-slave-wg14 krb5kdc[561](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781920957, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:02:38.930415 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40445
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:02:39.476332 874 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:02:39.476707 874 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:02:39.476827 874 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:02:39.490630 874 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:02:39.491118 874 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:02:39.491266 874 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:02:39.491354 874 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:02:39.505110 874 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:02:39.506965 874 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:02:39.508844 874 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:02:39.530506 880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:39.530462 882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:39.530563 879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:39.535665 874 server_base.cc:1061] running on GCE node
I20260620 02:02:39.537078 874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:02:39.539363 874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:02:39.541008 874 hybrid_clock.cc:648] HybridClock initialized: now 1781920959540788 us; error 105 us; skew 500 ppm
I20260620 02:02:39.541556 874 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:02:39 dist-test-slave-wg14 krb5kdc[561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920959, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:02:39.545766 874 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:02:39.548635 874 webserver.cc:533] Webserver started at http://127.20.1.190:36535/ using document root <none> and password file <none>
I20260620 02:02:39.549985 874 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:39.550164 874 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:39.550786 874 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:02:39.553946 874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c"
format_stamp: "Formatted at 2026-06-20 02:02:39 on dist-test-slave-wg14"
I20260620 02:02:39.555682 874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c"
format_stamp: "Formatted at 2026-06-20 02:02:39 on dist-test-slave-wg14"
I20260620 02:02:39.564632 874 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260620 02:02:39.570122 889 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:39.572031 874 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 02:02:39.572310 874 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c"
format_stamp: "Formatted at 2026-06-20 02:02:39 on dist-test-slave-wg14"
I20260620 02:02:39.572549 874 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:39.632872 874 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:02:39.643707 874 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:02:39.644438 874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:02:39.683233 874 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:40445
I20260620 02:02:39.683233 940 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:40445 every 8 connection(s)
I20260620 02:02:39.690446 874 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:02:39.698146 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:02:39.700044 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 874
I20260620 02:02:39.700263 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:02:39.707082 941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.189@KRBTEST.COM" created.
Jun 20 02:02:39 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920959, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:02:39.764395 941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } has no permanent_uuid. Determining permanent_uuid...
W20260620 02:02:39.766691 890 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:32931: connect: Connection refused (error 111)
W20260620 02:02:39.769447 941 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:32931: Network error: Client connection negotiation failed: client connection to 127.20.1.189:32931: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:02:39.808672 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.189: real 0.107s user 0.007s sys 0.017s
I20260620 02:02:39.823542 941 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } attempt: 1
W20260620 02:02:39.826422 941 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:32931: Network error: Client connection negotiation failed: client connection to 127.20.1.189:32931: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.20.1.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.189@KRBTEST.COM" created.
I20260620 02:02:39.894778 941 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } attempt: 2
W20260620 02:02:39.898077 941 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:32931: Network error: Client connection negotiation failed: client connection to 127.20.1.189:32931: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:02:39.903633 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:32931
--webserver_interface=127.20.1.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:02:39.989516 941 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } attempt: 3
W20260620 02:02:39.992412 941 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:32931: Network error: Client connection negotiation failed: client connection to 127.20.1.189:32931: connect: Connection refused (error 111)
I20260620 02:02:40.135744 941 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } attempt: 4
W20260620 02:02:40.139182 941 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:32931: Network error: Client connection negotiation failed: client connection to 127.20.1.189:32931: connect: Connection refused (error 111)
W20260620 02:02:40.429636 950 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:02:40.430073 950 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:02:40.430187 950 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260620 02:02:40.438637 941 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } attempt: 5
W20260620 02:02:40.442123 941 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:32931: Network error: Client connection negotiation failed: client connection to 127.20.1.189:32931: connect: Connection refused (error 111)
W20260620 02:02:40.443979 950 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:02:40.444118 950 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:02:40.444181 950 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:02:40.444235 950 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:02:40.462349 950 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:32931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:02:40.464489 950 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:02:40.466177 950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:02:40.485584 955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:40.485867 956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:40.487568 958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:40.489138 950 server_base.cc:1061] running on GCE node
I20260620 02:02:40.491727 950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:02:40.497071 950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:02:40.500936 950 hybrid_clock.cc:648] HybridClock initialized: now 1781920960500754 us; error 54 us; skew 500 ppm
I20260620 02:02:40.501580 950 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:02:40 dist-test-slave-wg14 krb5kdc[561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920960, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:02:40.506385 950 init.cc:377] Logged in from keytab as kudu/127.20.1.189@KRBTEST.COM (short username kudu)
I20260620 02:02:40.509105 950 webserver.cc:533] Webserver started at http://127.20.1.189:43191/ using document root <none> and password file <none>
I20260620 02:02:40.510068 950 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:40.510219 950 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:40.510653 950 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:02:40.515331 950 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "cd13271501f24e7bb8badec00d0aae5d"
format_stamp: "Formatted at 2026-06-20 02:02:40 on dist-test-slave-wg14"
I20260620 02:02:40.516305 950 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "cd13271501f24e7bb8badec00d0aae5d"
format_stamp: "Formatted at 2026-06-20 02:02:40 on dist-test-slave-wg14"
I20260620 02:02:40.528164 950 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.010s sys 0.000s
I20260620 02:02:40.542150 965 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:40.545297 950 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.007s sys 0.003s
I20260620 02:02:40.545616 950 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "cd13271501f24e7bb8badec00d0aae5d"
format_stamp: "Formatted at 2026-06-20 02:02:40 on dist-test-slave-wg14"
I20260620 02:02:40.545899 950 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:40.575311 950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:02:40.582700 950 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:02:40.583022 950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:02:40.636514 950 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:32931
I20260620 02:02:40.636518 1017 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:32931 every 8 connection(s)
I20260620 02:02:40.639555 950 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 02:02:40.646916 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 950
I20260620 02:02:40.647111 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 02:02:40.650647 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:02:40.664386 1018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:02:40 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920960, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.188@KRBTEST.COM" created.
I20260620 02:02:40.726864 1018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:02:40 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920960, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:02:40.757143 1018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } has no permanent_uuid. Determining permanent_uuid...
W20260620 02:02:40.759363 969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46379: connect: Connection refused (error 111)
W20260620 02:02:40.763015 1018 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46379: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46379: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:02:40.771704 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.188: real 0.124s user 0.007s sys 0.019s
WARNING: no policy specified for HTTP/127.20.1.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.188@KRBTEST.COM" created.
I20260620 02:02:40.813585 1018 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } attempt: 1
W20260620 02:02:40.816751 1018 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46379: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46379: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:02:40.879546 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.188: real 0.108s user 0.001s sys 0.019s
I20260620 02:02:40.880962 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:46379
--webserver_interface=127.20.1.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:02:40.885275 1018 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } attempt: 2
W20260620 02:02:40.890113 1018 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46379: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46379: connect: Connection refused (error 111)
I20260620 02:02:40.988884 1018 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } attempt: 3
I20260620 02:02:40.989473 941 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } attempt: 6
W20260620 02:02:40.992328 1018 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46379: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46379: connect: Connection refused (error 111)
Jun 20 02:02:41 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920959, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:02:41.032966 941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } has no permanent_uuid. Determining permanent_uuid...
W20260620 02:02:41.036783 941 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46379: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46379: connect: Connection refused (error 111)
I20260620 02:02:41.089178 941 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } attempt: 1
W20260620 02:02:41.092368 941 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46379: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46379: connect: Connection refused (error 111)
I20260620 02:02:41.135707 1018 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } attempt: 4
W20260620 02:02:41.139631 1018 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46379: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46379: connect: Connection refused (error 111)
I20260620 02:02:41.166868 941 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } attempt: 2
W20260620 02:02:41.170140 941 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46379: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46379: connect: Connection refused (error 111)
I20260620 02:02:41.283443 941 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } attempt: 3
W20260620 02:02:41.288059 941 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46379: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46379: connect: Connection refused (error 111)
I20260620 02:02:41.437484 941 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } attempt: 4
W20260620 02:02:41.443009 941 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46379: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46379: connect: Connection refused (error 111)
I20260620 02:02:41.442972 1018 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } attempt: 5
W20260620 02:02:41.448343 1018 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46379: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46379: connect: Connection refused (error 111)
W20260620 02:02:41.576452 1028 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:02:41.576814 1028 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:02:41.577112 1028 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:02:41.588349 1028 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:02:41.588492 1028 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:02:41.588560 1028 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:02:41.589354 1028 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:02:41.607324 1028 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.188:46379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.1.188
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:02:41.610265 1028 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:02:41.614739 1028 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:02:41.633435 1035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:41.634001 1038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:41.634066 1036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:41.638243 1028 server_base.cc:1061] running on GCE node
I20260620 02:02:41.645130 1028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:02:41.649828 1028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:02:41.657672 1028 hybrid_clock.cc:648] HybridClock initialized: now 1781920961657440 us; error 132 us; skew 500 ppm
I20260620 02:02:41.658241 1028 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:02:41 dist-test-slave-wg14 krb5kdc[561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920961, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:02:41.669405 1028 init.cc:377] Logged in from keytab as kudu/127.20.1.188@KRBTEST.COM (short username kudu)
I20260620 02:02:41.672154 1028 webserver.cc:533] Webserver started at http://127.20.1.188:39709/ using document root <none> and password file <none>
I20260620 02:02:41.673282 1028 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:41.673475 1028 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:41.673985 1028 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:02:41.677759 1028 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-2/data/instance:
uuid: "58dfca13e8144471a3687f2ca3e8463c"
format_stamp: "Formatted at 2026-06-20 02:02:41 on dist-test-slave-wg14"
I20260620 02:02:41.679075 1028 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-2/wal/instance:
uuid: "58dfca13e8144471a3687f2ca3e8463c"
format_stamp: "Formatted at 2026-06-20 02:02:41 on dist-test-slave-wg14"
I20260620 02:02:41.694289 1028 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.005s sys 0.004s
I20260620 02:02:41.699932 1045 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:41.706468 1028 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.008s sys 0.000s
I20260620 02:02:41.706734 1028 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-2/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-2/wal
uuid: "58dfca13e8144471a3687f2ca3e8463c"
format_stamp: "Formatted at 2026-06-20 02:02:41 on dist-test-slave-wg14"
I20260620 02:02:41.706977 1028 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:41.711371 941 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } attempt: 5
W20260620 02:02:41.713836 941 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.188:46379: Network error: Client connection negotiation failed: client connection to 127.20.1.188:46379: connect: Connection refused (error 111)
I20260620 02:02:41.738497 1028 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:02:41.753486 1028 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:02:41.753847 1028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:02:41.815389 1096 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.188:46379 every 8 connection(s)
I20260620 02:02:41.815411 1028 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.188:46379
I20260620 02:02:41.818324 1028 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-2/data/info.pb
I20260620 02:02:41.829764 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 1028
I20260620 02:02:41.830015 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-2/wal/instance
I20260620 02:02:41.835724 1097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:02:41 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:02:41.856204 1097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:02:41 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 02:02:41 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920961, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:02:41.907480 1097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:02:41 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920961, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 02:02:41 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:02:41.935391 1097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:02:41 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920961, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.188@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:02:41.982476 1097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c: Bootstrap starting.
I20260620 02:02:41.991863 1097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c: Neither blocks nor log segments found. Creating new log.
I20260620 02:02:41.995656 1018 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } attempt: 6
I20260620 02:02:41.996595 1097 log.cc:826] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:02:42.001466 1097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c: No bootstrap required, opened a new log
I20260620 02:02:42.014145 1097 raft_consensus.cc:359] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } }
I20260620 02:02:42.014874 1097 raft_consensus.cc:385] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:02:42.015030 1097 raft_consensus.cc:740] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dfca13e8144471a3687f2ca3e8463c, State: Initialized, Role: FOLLOWER
I20260620 02:02:42.015894 1097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } }
I20260620 02:02:42.020649 1108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } }
I20260620 02:02:42.021282 1108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:42.022241 1097 sys_catalog.cc:565] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 02:02:42 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920960, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
W20260620 02:02:42.052456 1119 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:02:42.053012 1119 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:02:42.034211 1028 master_runner.cc:187] Error getting master registration for 127.20.1.190:40445: OK, instance_id { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" instance_seqno: 1781920959674597 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:02:42.053586 1028 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:02:42.063863 1018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d: Bootstrap starting.
I20260620 02:02:42.074354 1018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d: Neither blocks nor log segments found. Creating new log.
I20260620 02:02:42.077018 1018 log.cc:826] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d: Log is configured to *not* fsync() on all Append() calls
I20260620 02:02:42.095634 1018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d: No bootstrap required, opened a new log
I20260620 02:02:42.105001 1018 raft_consensus.cc:359] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } }
I20260620 02:02:42.105448 1018 raft_consensus.cc:385] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:02:42.105595 1018 raft_consensus.cc:740] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd13271501f24e7bb8badec00d0aae5d, State: Initialized, Role: FOLLOWER
I20260620 02:02:42.106719 1018 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } }
I20260620 02:02:42.110881 1122 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } }
I20260620 02:02:42.112206 1122 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:42.114264 1018 sys_catalog.cc:565] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:02:42.152577 1133 catalog_manager.cc:1593] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:02:42.152784 1133 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:02:42.135560 950 master_runner.cc:187] Error getting master registration for 127.20.1.190:40445: OK, instance_id { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" instance_seqno: 1781920959674597 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:02:42.156388 950 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:02:42.222947 1122 raft_consensus.cc:493] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:02:42.223573 1122 raft_consensus.cc:515] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } }
I20260620 02:02:42.242341 1122 leader_election.cc:290] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3593436f1f74fb7ad9fb1fe9b642b0c (127.20.1.190:40445), 58dfca13e8144471a3687f2ca3e8463c (127.20.1.188:46379)
I20260620 02:02:42.245596 1072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd13271501f24e7bb8badec00d0aae5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dfca13e8144471a3687f2ca3e8463c" is_pre_election: true
I20260620 02:02:42.251901 1072 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd13271501f24e7bb8badec00d0aae5d in term 0.
I20260620 02:02:42.253108 941 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } attempt: 6
I20260620 02:02:42.253161 969 leader_election.cc:304] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58dfca13e8144471a3687f2ca3e8463c, cd13271501f24e7bb8badec00d0aae5d; no voters:
I20260620 02:02:42.254899 1122 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:02:42.255277 1122 raft_consensus.cc:493] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:02:42.255569 1122 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [term 0 FOLLOWER]: Advancing to term 1
W20260620 02:02:42.256836 1051 tablet.cc:2406] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:02:42.269495 1122 raft_consensus.cc:515] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } }
I20260620 02:02:42.272066 1122 leader_election.cc:290] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [CANDIDATE]: Term 1 election: Requested vote from peers e3593436f1f74fb7ad9fb1fe9b642b0c (127.20.1.190:40445), 58dfca13e8144471a3687f2ca3e8463c (127.20.1.188:46379)
I20260620 02:02:42.274042 1072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd13271501f24e7bb8badec00d0aae5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dfca13e8144471a3687f2ca3e8463c"
I20260620 02:02:42.274355 1072 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:02:42.299533 1072 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd13271501f24e7bb8badec00d0aae5d in term 1.
I20260620 02:02:42.305013 969 leader_election.cc:304] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58dfca13e8144471a3687f2ca3e8463c, cd13271501f24e7bb8badec00d0aae5d; no voters:
I20260620 02:02:42.306241 1122 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:02:42.312474 1122 raft_consensus.cc:697] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [term 1 LEADER]: Becoming Leader. State: Replica: cd13271501f24e7bb8badec00d0aae5d, State: Running, Role: LEADER
Jun 20 02:02:42 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920959, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:02:42.315097 1122 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } }
I20260620 02:02:42.330536 1136 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd13271501f24e7bb8badec00d0aae5d. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } }
I20260620 02:02:42.331226 1136 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: This master's current role is: LEADER
I20260620 02:02:42.339607 1139 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:02:42.340620 941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c: Bootstrap starting.
W20260620 02:02:42.341342 972 tablet.cc:2406] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:02:42.346056 1139 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:02:42.359596 941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c: Neither blocks nor log segments found. Creating new log.
I20260620 02:02:42.361804 941 log.cc:826] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:02:42.374439 941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c: No bootstrap required, opened a new log
I20260620 02:02:42.389611 1072 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [term 1 FOLLOWER]: Refusing update from remote peer cd13271501f24e7bb8badec00d0aae5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:02:42.391350 1122 consensus_queue.cc:1048] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 02:02:42.412436 941 raft_consensus.cc:359] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } }
I20260620 02:02:42.413414 941 raft_consensus.cc:385] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:02:42.413630 941 raft_consensus.cc:740] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3593436f1f74fb7ad9fb1fe9b642b0c, State: Initialized, Role: FOLLOWER
I20260620 02:02:42.419337 941 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } }
I20260620 02:02:42.422465 1144 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } }
I20260620 02:02:42.422824 1144 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:42.425200 941 sys_catalog.cc:565] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:02:42.425686 914 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:02:42.430162 915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd13271501f24e7bb8badec00d0aae5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c"
I20260620 02:02:42.431517 916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd13271501f24e7bb8badec00d0aae5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" is_pre_election: true
I20260620 02:02:42.435281 1108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd13271501f24e7bb8badec00d0aae5d. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } }
I20260620 02:02:42.435774 1108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:42.436524 914 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [term 1 FOLLOWER]: Refusing update from remote peer cd13271501f24e7bb8badec00d0aae5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:02:42.436848 1141 mvcc.cc:204] Tried to move back new op lower bound from 7298748261931720704 to 7298748261649911808. Current Snapshot: MvccSnapshot[applied={T|T < 7298748261931720704}]
I20260620 02:02:42.439742 1136 consensus_queue.cc:1048] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:02:42.446291 1108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } }
I20260620 02:02:42.446638 1108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:42.454910 1136 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } }
I20260620 02:02:42.455250 1136 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: This master's current role is: LEADER
I20260620 02:02:42.459035 1142 mvcc.cc:204] Tried to move back new op lower bound from 7298748261931720704 to 7298748261649911808. Current Snapshot: MvccSnapshot[applied={T|T < 7298748261931720704}]
I20260620 02:02:42.479172 1139 catalog_manager.cc:1382] Generated new cluster ID: e29bc8003431478491db0527118fd622
I20260620 02:02:42.479383 1139 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:02:42.545457 1160 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } }
I20260620 02:02:42.545868 1160 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:42.546391 1162 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:02:42.546355 1144 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd13271501f24e7bb8badec00d0aae5d. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } }
I20260620 02:02:42.546626 1144 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:42.550740 1122 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } }
I20260620 02:02:42.551066 1122 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: This master's current role is: LEADER
I20260620 02:02:42.555403 1139 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:02:42.559512 1139 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:02:42.592207 1139 catalog_manager.cc:6069] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d: Generated new TSK 0
I20260620 02:02:42.598774 1139 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:02:42.732434 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.125s user 0.006s sys 0.024s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:02:42.827926 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:02:43.056717 1119 catalog_manager.cc:1294] Loaded cluster ID: e29bc8003431478491db0527118fd622
I20260620 02:02:43.056867 1119 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c: loading cluster ID for follower catalog manager: success
I20260620 02:02:43.061741 1119 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c: acquiring CA information for follower catalog manager: success
I20260620 02:02:43.065446 1119 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 02:02:43.439492 1167 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:02:43.440021 1167 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:02:43.440169 1167 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:02:43.450130 1167 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:02:43.450433 1167 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:02:43.465726 1167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:02:43.467756 1167 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:02:43.470058 1167 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:02:43.490422 1173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:43.491310 1175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:43.490515 1172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:43.495404 1167 server_base.cc:1061] running on GCE node
I20260620 02:02:43.496333 1167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:02:43.497865 1167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:02:43.501456 1167 hybrid_clock.cc:648] HybridClock initialized: now 1781920963501231 us; error 141 us; skew 500 ppm
I20260620 02:02:43.501960 1167 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:02:43 dist-test-slave-wg14 krb5kdc[561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920963, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:02:43.506276 1167 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:02:43.509079 1167 webserver.cc:533] Webserver started at http://127.20.1.129:35649/ using document root <none> and password file <none>
I20260620 02:02:43.511358 1167 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:43.511775 1167 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:43.513466 1167 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:02:43.517143 1167 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "053b7949c7f84aab8620669b9d8d9786"
format_stamp: "Formatted at 2026-06-20 02:02:43 on dist-test-slave-wg14"
I20260620 02:02:43.519727 1167 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "053b7949c7f84aab8620669b9d8d9786"
format_stamp: "Formatted at 2026-06-20 02:02:43 on dist-test-slave-wg14"
I20260620 02:02:43.528818 1167 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260620 02:02:43.537477 1182 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:43.540252 1167 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260620 02:02:43.540566 1167 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "053b7949c7f84aab8620669b9d8d9786"
format_stamp: "Formatted at 2026-06-20 02:02:43 on dist-test-slave-wg14"
I20260620 02:02:43.540841 1167 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:43.553423 1162 catalog_manager.cc:1294] Loaded cluster ID: e29bc8003431478491db0527118fd622
I20260620 02:02:43.553588 1162 catalog_manager.cc:1587] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c: loading cluster ID for follower catalog manager: success
I20260620 02:02:43.558780 1162 catalog_manager.cc:1609] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c: acquiring CA information for follower catalog manager: success
I20260620 02:02:43.566445 1162 catalog_manager.cc:1637] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:02:43.582626 1167 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:02:43.587972 1167 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:02:43.588513 1167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:02:43.592388 1167 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:02:43.595618 1167 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:02:43.595758 1167 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:43.596050 1167 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:02:43.596148 1167 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:43.717209 1167 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:39129
I20260620 02:02:43.717252 1298 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:39129 every 8 connection(s)
I20260620 02:02:43.719159 1167 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:02:43.728320 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 1167
I20260620 02:02:43.728576 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 02:02:43 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920963, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 02:02:43 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920963, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
Jun 20 02:02:43 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920963, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:02:43.830998 1303 heartbeater.cc:344] Connected to a master server at 127.20.1.189:32931
I20260620 02:02:43.833379 1303 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:43.837872 1300 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40445
I20260620 02:02:43.838732 1300 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:43.845111 1303 heartbeater.cc:507] Master 127.20.1.189:32931 requested a full tablet report, sending...
I20260620 02:02:43.845419 1300 heartbeater.cc:507] Master 127.20.1.190:40445 requested a full tablet report, sending...
I20260620 02:02:43.851290 1299 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46379
I20260620 02:02:43.851703 1299 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:43.852427 1299 heartbeater.cc:507] Master 127.20.1.188:46379 requested a full tablet report, sending...
I20260620 02:02:43.853648 983 ts_manager.cc:194] Registered new tserver with Master: 053b7949c7f84aab8620669b9d8d9786 (127.20.1.129:39129)
I20260620 02:02:43.855875 983 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:46343
I20260620 02:02:43.858470 1061 ts_manager.cc:194] Registered new tserver with Master: 053b7949c7f84aab8620669b9d8d9786 (127.20.1.129:39129)
I20260620 02:02:43.858676 905 ts_manager.cc:194] Registered new tserver with Master: 053b7949c7f84aab8620669b9d8d9786 (127.20.1.129:39129)
I20260620 02:02:43.879247 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
WARNING: no policy specified for kudu/127.20.1.187@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.187@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.187 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.187 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.187@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.187@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.187 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.187 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:02:44.076746 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:37329
--webserver_interface=127.20.1.187
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.187
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379,127.20.1.187:37329 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:02:44.710530 1313 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:02:44.710994 1313 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:02:44.711181 1313 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:02:44.727025 1313 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:02:44.727205 1313 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:02:44.727278 1313 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:02:44.727347 1313 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:02:44.742375 1313 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379,127.20.1.187:37329
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.187
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.187:37329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.20.1.187
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:02:44.744385 1313 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:02:44.746202 1313 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:02:44.761009 1319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:44.761076 1318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:44.761982 1321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:44.762307 1313 server_base.cc:1061] running on GCE node
I20260620 02:02:44.763401 1313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:02:44.765566 1313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:02:44.769271 1313 hybrid_clock.cc:648] HybridClock initialized: now 1781920964769046 us; error 122 us; skew 500 ppm
I20260620 02:02:44.770092 1313 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:02:44 dist-test-slave-wg14 krb5kdc[561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920964, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:02:44.778698 1313 init.cc:377] Logged in from keytab as kudu/127.20.1.187@KRBTEST.COM (short username kudu)
I20260620 02:02:44.781178 1313 webserver.cc:533] Webserver started at http://127.20.1.187:33797/ using document root <none> and password file <none>
I20260620 02:02:44.782141 1313 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:44.782289 1313 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:44.782713 1313 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:02:44.785873 1313 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/data/instance:
uuid: "75c0012122af495a95fffc7a92fc22a4"
format_stamp: "Formatted at 2026-06-20 02:02:44 on dist-test-slave-wg14"
I20260620 02:02:44.787066 1313 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/wal/instance:
uuid: "75c0012122af495a95fffc7a92fc22a4"
format_stamp: "Formatted at 2026-06-20 02:02:44 on dist-test-slave-wg14"
I20260620 02:02:44.801304 1313 fs_manager.cc:696] Time spent creating directory manager: real 0.014s user 0.001s sys 0.011s
I20260620 02:02:44.812656 1328 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:44.815918 1313 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260620 02:02:44.816231 1313 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "75c0012122af495a95fffc7a92fc22a4"
format_stamp: "Formatted at 2026-06-20 02:02:44 on dist-test-slave-wg14"
I20260620 02:02:44.816499 1313 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:44.858803 1313 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:02:44.865041 1303 heartbeater.cc:499] Master 127.20.1.189:32931 was elected leader, sending a full tablet report...
I20260620 02:02:44.871872 1313 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:02:44.872591 1313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:02:44.933238 1313 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:37329
I20260620 02:02:44.933334 1379 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:37329 every 8 connection(s)
I20260620 02:02:44.937806 1313 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 02:02:44.944308 1380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:02:44.945793 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 1313
I20260620 02:02:44.954665 1380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:02:44 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
Jun 20 02:02:44 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920964, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:02:44.988236 1380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:02:45 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920964, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:02:45.016356 1380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:02:45 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920964, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
I20260620 02:02:45.041186 1380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:02:45 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920964, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
I20260620 02:02:45.066692 1380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: Bootstrap starting.
I20260620 02:02:45.071583 1380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: Neither blocks nor log segments found. Creating new log.
I20260620 02:02:45.074172 1380 log.cc:826] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: Log is configured to *not* fsync() on all Append() calls
I20260620 02:02:45.079463 1380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: No bootstrap required, opened a new log
I20260620 02:02:45.088243 1380 raft_consensus.cc:359] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } }
I20260620 02:02:45.088830 1380 raft_consensus.cc:385] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:02:45.088958 1380 raft_consensus.cc:740] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c0012122af495a95fffc7a92fc22a4, State: Initialized, Role: FOLLOWER
I20260620 02:02:45.090145 1380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } }
I20260620 02:02:45.091467 1389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } } }
I20260620 02:02:45.091915 1389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:45.093393 1380 sys_catalog.cc:565] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:02:45.107034 1401 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:02:45.107265 1401 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:02:45.122592 1313 master_runner.cc:429] Detected that this master 75c0012122af495a95fffc7a92fc22a4 is joining an existing cluster
I20260620 02:02:45.122782 1313 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260620 02:02:45.124089 1313 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 02:02:45.124631 1313 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 02:02:45.124902 1313 tablet_replica.cc:333] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: stopping tablet replica
I20260620 02:02:45.128767 1313 master_runner.cc:434] Initiating AddMaster RPC to add 127.20.1.187:37329
Jun 20 02:02:45 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920960, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.189@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
I20260620 02:02:45.270534 983 catalog_manager.cc:7201] Initiating ChangeConfig request to add master 127.20.1.187:37329: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "75c0012122af495a95fffc7a92fc22a4"
member_type: NON_VOTER
last_known_addr {
host: "127.20.1.187"
port: 37329
}
attrs {
promote: true
}
}
dest_uuid: "cd13271501f24e7bb8badec00d0aae5d"
cas_config_opid_index: -1
I20260620 02:02:45.272105 983 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: true } }
I20260620 02:02:45.279642 916 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [term 1 FOLLOWER]: Refusing update from remote peer cd13271501f24e7bb8badec00d0aae5d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 02:02:45.280866 1414 consensus_queue.cc:1048] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260620 02:02:45.281679 968 consensus_peers.cc:597] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d -> Peer 75c0012122af495a95fffc7a92fc22a4 (127.20.1.187:37329): Couldn't send request to peer 75c0012122af495a95fffc7a92fc22a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260620 02:02:45.283370 1071 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [term 1 FOLLOWER]: Refusing update from remote peer cd13271501f24e7bb8badec00d0aae5d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 02:02:45.284855 1414 consensus_queue.cc:1048] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 02:02:45.287519 1192 raft_consensus.cc:2955] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 75c0012122af495a95fffc7a92fc22a4 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: true } } }
I20260620 02:02:45.295903 916 raft_consensus.cc:2955] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 75c0012122af495a95fffc7a92fc22a4 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: true } } }
I20260620 02:02:45.299602 1192 catalog_manager.cc:7205] Successfully completed master ChangeConfig request to add master 127.20.1.187:37329
I20260620 02:02:45.305161 1313 master.cc:562] Master@127.20.1.187:37329 shutting down...
I20260620 02:02:45.305958 1071 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 75c0012122af495a95fffc7a92fc22a4 (127.20.1.187) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: true } } }
I20260620 02:02:45.306579 1419 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: true } } }
I20260620 02:02:45.306995 1419 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:45.307761 1192 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: true } } }
I20260620 02:02:45.307813 1417 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: true } } }
I20260620 02:02:45.308279 1192 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: This master's current role is: LEADER
I20260620 02:02:45.308311 1417 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: This master's current role is: LEADER
I20260620 02:02:45.308663 1414 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: true } } }
I20260620 02:02:45.310938 1414 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: This master's current role is: LEADER
I20260620 02:02:45.338511 1422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: true } } }
I20260620 02:02:45.339196 1422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:45.354041 1313 master.cc:584] Master@127.20.1.187:37329 shutdown complete.
I20260620 02:02:45.354146 1313 master_runner.cc:306] Clearing existing system tablet
I20260620 02:02:45.354662 1313 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:45.354801 1313 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:45.359894 1313 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 02:02:45.362900 1424 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:45.363595 1313 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:45.363902 1313 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:02:45.364046 1313 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "75c0012122af495a95fffc7a92fc22a4"
format_stamp: "Formatted at 2026-06-20 02:02:44 on dist-test-slave-wg14"
I20260620 02:02:45.370527 1313 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:02:45.380066 1313 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 02:02:45.380291 1313 log.cc:1199] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: Deleting WAL directory at /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260620 02:02:45.381008 1313 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: Deleting consensus metadata
I20260620 02:02:45.381929 1313 master_runner.cc:316] Copying system tablet from 127.20.1.189:32931
I20260620 02:02:45.387171 1313 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.1.189:32931
I20260620 02:02:45.410306 1003 tablet_copy_service.cc:140] P cd13271501f24e7bb8badec00d0aae5d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 75c0012122af495a95fffc7a92fc22a4 ({username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:58396)
I20260620 02:02:45.410650 1003 tablet_copy_service.cc:161] P cd13271501f24e7bb8badec00d0aae5d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 75c0012122af495a95fffc7a92fc22a4 at {username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:58396: session id = 75c0012122af495a95fffc7a92fc22a4-00000000000000000000000000000000
I20260620 02:02:45.414423 1003 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 02:02:45.418267 1313 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 02:02:45.425908 1313 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: tablet copy: Starting download of 0 data blocks...
I20260620 02:02:45.426303 1313 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: tablet copy: Starting download of 1 WAL segments...
I20260620 02:02:45.432374 1313 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 02:02:45.442046 1003 tablet_copy_service.cc:342] P cd13271501f24e7bb8badec00d0aae5d: Request end of tablet copy session 75c0012122af495a95fffc7a92fc22a4-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.20.1.187@KRBTEST.COM'} at 127.0.0.1:58396
I20260620 02:02:45.442404 1003 tablet_copy_service.cc:434] P cd13271501f24e7bb8badec00d0aae5d: ending tablet copy session 75c0012122af495a95fffc7a92fc22a4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 75c0012122af495a95fffc7a92fc22a4
W20260620 02:02:45.462471 1313 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260620 02:02:45.473980 1313 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:02:45.477907 1431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:45.479393 1432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:45.485572 1313 server_base.cc:1061] running on GCE node
W20260620 02:02:45.485723 1434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:45.486269 1313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:02:45.487345 1313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:02:45.488611 1313 hybrid_clock.cc:648] HybridClock initialized: now 1781920965488507 us; error 62 us; skew 500 ppm
I20260620 02:02:45.489583 1313 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:02:45 dist-test-slave-wg14 krb5kdc[561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920965, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.187@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:02:45.499176 1313 init.cc:377] Logged in from keytab as kudu/127.20.1.187@KRBTEST.COM (short username kudu)
I20260620 02:02:45.501093 1313 webserver.cc:533] Webserver started at http://127.20.1.187:34143/ using document root <none> and password file <none>
I20260620 02:02:45.501601 1313 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:45.501727 1313 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:45.505431 1313 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 02:02:45.508342 1441 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:45.508934 1313 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 02:02:45.510341 1313 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/wal
uuid: "75c0012122af495a95fffc7a92fc22a4"
format_stamp: "Formatted at 2026-06-20 02:02:44 on dist-test-slave-wg14"
I20260620 02:02:45.510630 1313 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:45.553260 1313 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:02:45.561371 1313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:02:45.595850 1313 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.187:37329
I20260620 02:02:45.595875 1493 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.187:37329 every 8 connection(s)
I20260620 02:02:45.597699 1313 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-3/data/info.pb
I20260620 02:02:45.606796 1494 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:02:45.615542 1494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: Bootstrap starting.
I20260620 02:02:45.658375 1494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:02:45.659711 1494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: Bootstrap complete.
I20260620 02:02:45.661262 1494 raft_consensus.cc:359] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: true } }
I20260620 02:02:45.661526 1494 raft_consensus.cc:740] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 75c0012122af495a95fffc7a92fc22a4, State: Initialized, Role: LEARNER
I20260620 02:02:45.661815 1494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: true } }
I20260620 02:02:45.663798 1494 sys_catalog.cc:565] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:02:45.663549 1498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: true } } }
I20260620 02:02:45.664143 1498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:02:45.685869 1509 catalog_manager.cc:1294] Loaded cluster ID: e29bc8003431478491db0527118fd622
I20260620 02:02:45.686024 1509 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: loading cluster ID for follower catalog manager: success
I20260620 02:02:45.700402 1509 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: acquiring CA information for follower catalog manager: success
I20260620 02:02:45.703775 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 1167
I20260620 02:02:45.704711 1509 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Jun 20 02:02:45 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:02:45.736864 1293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:02:45.850497 1468 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260620 02:02:45.851564 1498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd13271501f24e7bb8badec00d0aae5d. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: true } } }
I20260620 02:02:45.851851 1498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:02:45.921360 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 1167
W20260620 02:02:45.942059 969 connection.cc:573] server connection from 127.20.1.129:46343 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 02:02:45.942092 891 connection.cc:573] server connection from 127.20.1.129:55427 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:02:45.943526 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:39129
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379
--webserver_port=35649
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379,127.20.1.187:37329 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:02:46.338969 1192 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: NON_VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: true } } }
I20260620 02:02:46.339376 1192 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: This master's current role is: LEADER
I20260620 02:02:46.341418 1514 raft_consensus.cc:1064] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d: attempting to promote NON_VOTER 75c0012122af495a95fffc7a92fc22a4 to VOTER
I20260620 02:02:46.342592 1514 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } }
I20260620 02:02:46.346218 1071 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [term 1 FOLLOWER]: Refusing update from remote peer cd13271501f24e7bb8badec00d0aae5d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:02:46.346796 916 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [term 1 FOLLOWER]: Refusing update from remote peer cd13271501f24e7bb8badec00d0aae5d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:02:46.348932 1468 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [term 1 LEARNER]: Refusing update from remote peer cd13271501f24e7bb8badec00d0aae5d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:02:46.350487 1192 consensus_queue.cc:1048] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:02:46.351950 1519 consensus_queue.cc:1048] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:02:46.353401 1519 consensus_queue.cc:1048] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:02:46.367326 1518 raft_consensus.cc:2955] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 75c0012122af495a95fffc7a92fc22a4 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } } }
I20260620 02:02:46.368558 1468 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 75c0012122af495a95fffc7a92fc22a4 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } } }
I20260620 02:02:46.369805 916 raft_consensus.cc:2955] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 75c0012122af495a95fffc7a92fc22a4 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } } }
I20260620 02:02:46.381956 1071 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 75c0012122af495a95fffc7a92fc22a4 (127.20.1.187) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } } }
I20260620 02:02:46.388041 1515 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } } }
I20260620 02:02:46.388409 1515 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: This master's current role is: LEADER
I20260620 02:02:46.388005 1523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } } }
I20260620 02:02:46.388914 1523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75c0012122af495a95fffc7a92fc22a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:46.389063 1514 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } } }
I20260620 02:02:46.389284 1514 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: This master's current role is: LEADER
I20260620 02:02:46.390627 1524 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } } }
I20260620 02:02:46.391023 1524 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:46.391050 1525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } } }
I20260620 02:02:46.391396 1525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dfca13e8144471a3687f2ca3e8463c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:46.391772 1514 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } } }
I20260620 02:02:46.391867 1515 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } } }
I20260620 02:02:46.392129 1514 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: This master's current role is: LEADER
I20260620 02:02:46.392164 1515 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: This master's current role is: LEADER
W20260620 02:02:46.548085 1513 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:02:46.548478 1513 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:02:46.548641 1513 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:02:46.561221 1513 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:02:46.561455 1513 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:02:46.577623 1513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:39129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=35649
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379,127.20.1.187:37329
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:02:46.580132 1513 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:02:46.582329 1513 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:02:46.604907 1530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:46.607988 1531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:46.611528 1533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:46.612954 1513 server_base.cc:1061] running on GCE node
I20260620 02:02:46.613931 1513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:02:46.615762 1513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:02:46.619513 1513 hybrid_clock.cc:648] HybridClock initialized: now 1781920966619244 us; error 171 us; skew 500 ppm
I20260620 02:02:46.620329 1513 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:02:46 dist-test-slave-wg14 krb5kdc[561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920966, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:02:46.628583 1513 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:02:46.633226 1513 webserver.cc:533] Webserver started at http://127.20.1.129:35649/ using document root <none> and password file <none>
I20260620 02:02:46.634502 1513 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:46.634706 1513 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:46.643970 1513 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:02:46.648536 1540 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:46.653545 1513 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260620 02:02:46.653842 1513 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "053b7949c7f84aab8620669b9d8d9786"
format_stamp: "Formatted at 2026-06-20 02:02:43 on dist-test-slave-wg14"
I20260620 02:02:46.654912 1513 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:46.735103 1513 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:02:46.743503 1513 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:02:46.743872 1513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:02:46.745333 1513 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:02:46.747812 1513 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:02:46.748005 1513 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:46.748147 1513 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:02:46.748235 1513 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:46.808194 1513 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:39129
I20260620 02:02:46.808264 1653 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:39129 every 8 connection(s)
I20260620 02:02:46.812052 1513 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:02:46.817057 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 1513
Jun 20 02:02:46 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920966, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
Jun 20 02:02:46 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920966, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
Jun 20 02:02:46 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920966, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 02:02:46 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920966, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:02:46.891183 1654 heartbeater.cc:344] Connected to a master server at 127.20.1.187:37329
I20260620 02:02:46.891796 1654 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:46.892479 1655 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46379
I20260620 02:02:46.893034 1657 heartbeater.cc:344] Connected to a master server at 127.20.1.189:32931
I20260620 02:02:46.893213 1655 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:46.893425 1657 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:46.893823 1656 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40445
I20260620 02:02:46.894055 1656 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:46.894753 1656 heartbeater.cc:507] Master 127.20.1.190:40445 requested a full tablet report, sending...
I20260620 02:02:46.895267 1657 heartbeater.cc:507] Master 127.20.1.189:32931 requested a full tablet report, sending...
I20260620 02:02:46.895310 1654 heartbeater.cc:507] Master 127.20.1.187:37329 requested a full tablet report, sending...
I20260620 02:02:46.895658 1655 heartbeater.cc:507] Master 127.20.1.188:46379 requested a full tablet report, sending...
I20260620 02:02:46.897833 905 ts_manager.cc:194] Re-registered known tserver with Master: 053b7949c7f84aab8620669b9d8d9786 (127.20.1.129:39129)
I20260620 02:02:46.898279 1458 ts_manager.cc:194] Registered new tserver with Master: 053b7949c7f84aab8620669b9d8d9786 (127.20.1.129:39129)
I20260620 02:02:46.899698 1061 ts_manager.cc:194] Re-registered known tserver with Master: 053b7949c7f84aab8620669b9d8d9786 (127.20.1.129:39129)
I20260620 02:02:46.899758 983 ts_manager.cc:194] Re-registered known tserver with Master: 053b7949c7f84aab8620669b9d8d9786 (127.20.1.129:39129)
I20260620 02:02:46.901468 983 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:56423
I20260620 02:02:46.901618 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 02:02:46.920032 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 02:02:46.920236 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 874
I20260620 02:02:46.921504 936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:02:47.072271 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 874
I20260620 02:02:47.096683 1028 master_runner.cc:217] Error getting master consensus for 127.20.1.190:40445: Network error: recv got EOF from 127.20.1.190:40445 (error 108)
I20260620 02:02:47.097357 1028 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:02:47.098469 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40445
--webserver_interface=127.20.1.190
--webserver_port=36535
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379
--master_addresses=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379,127.20.1.187:37329 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:02:47.158259 950 master_runner.cc:187] Error getting master registration for 127.20.1.190:40445: Network error: Client connection negotiation failed: client connection to 127.20.1.190:40445: connect: Connection refused (error 111),
I20260620 02:02:47.158536 950 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260620 02:02:47.304111 967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.190:40445: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20260620 02:02:47.305745 967 consensus_peers.cc:597] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d -> Peer e3593436f1f74fb7ad9fb1fe9b642b0c (127.20.1.190:40445): Couldn't send request to peer e3593436f1f74fb7ad9fb1fe9b642b0c. Status: Network error: Client connection negotiation failed: client connection to 127.20.1.190:40445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260620 02:02:47.674368 1668 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:02:47.675663 1668 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:02:47.675837 1668 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:02:47.692320 1668 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:02:47.692494 1668 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:02:47.692553 1668 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:02:47.692606 1668 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:02:47.713372 1668 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379,127.20.1.187:37329
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=36535
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:02:47.715592 1668 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:02:47.718248 1668 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:02:47.736696 1675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:47.738080 1676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:47.738101 1678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:47.743144 1668 server_base.cc:1061] running on GCE node
I20260620 02:02:47.747959 1668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:02:47.751109 1668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:02:47.752854 1668 hybrid_clock.cc:648] HybridClock initialized: now 1781920967752633 us; error 65 us; skew 500 ppm
I20260620 02:02:47.753460 1668 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:02:47 dist-test-slave-wg14 krb5kdc[561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920967, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:02:47.757611 1668 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:02:47.761278 1668 webserver.cc:533] Webserver started at http://127.20.1.190:36535/ using document root <none> and password file <none>
I20260620 02:02:47.770248 1668 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:47.770449 1668 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:47.778007 1668 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:02:47.790040 1685 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:47.792870 1668 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.004s sys 0.000s
I20260620 02:02:47.793449 1668 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c"
format_stamp: "Formatted at 2026-06-20 02:02:39 on dist-test-slave-wg14"
I20260620 02:02:47.794533 1668 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:47.838284 1668 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:02:47.848420 1668 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:02:47.848743 1668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:02:47.887187 1514 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } } }
I20260620 02:02:47.887657 1514 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: This master's current role is: LEADER
I20260620 02:02:47.906807 1657 heartbeater.cc:499] Master 127.20.1.189:32931 was elected leader, sending a full tablet report...
W20260620 02:02:47.908627 1656 heartbeater.cc:646] Failed to heartbeat to 127.20.1.190:40445 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.1.190:40445: connect: Connection refused (error 111)
I20260620 02:02:47.910305 1668 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:40445
I20260620 02:02:47.910352 1744 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:40445 every 8 connection(s)
I20260620 02:02:47.915989 1668 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:02:47.920583 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 1668
I20260620 02:02:47.920908 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 1513
I20260620 02:02:47.952059 1747 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:02:47.957974 1747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c: Bootstrap starting.
I20260620 02:02:47.990145 1649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:02:47.997555 1747 log.cc:826] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:02:48.032912 1747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:02:48.035349 1747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c: Bootstrap complete.
I20260620 02:02:48.050462 1747 raft_consensus.cc:359] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } }
I20260620 02:02:48.052572 1747 raft_consensus.cc:740] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3593436f1f74fb7ad9fb1fe9b642b0c, State: Initialized, Role: FOLLOWER
I20260620 02:02:48.053531 1747 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } }
I20260620 02:02:48.055214 1753 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } } }
I20260620 02:02:48.055739 1753 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:48.056780 1747 sys_catalog.cc:565] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:02:48.057937 1753 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd13271501f24e7bb8badec00d0aae5d. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } } }
I20260620 02:02:48.058357 1753 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:02:48.097183 1764 catalog_manager.cc:1294] Loaded cluster ID: e29bc8003431478491db0527118fd622
I20260620 02:02:48.097358 1764 catalog_manager.cc:1587] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c: loading cluster ID for follower catalog manager: success
Jun 20 02:02:48 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920967, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:02:48.114002 1764 catalog_manager.cc:1609] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c: acquiring CA information for follower catalog manager: success
I20260620 02:02:48.118826 1764 catalog_manager.cc:1637] T 00000000000000000000000000000000 P e3593436f1f74fb7ad9fb1fe9b642b0c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:02:48.248191 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 1513
I20260620 02:02:48.289494 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:39129
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379
--webserver_port=35649
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379,127.20.1.187:37329 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:02:48.375604 1736 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd13271501f24e7bb8badec00d0aae5d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3593436f1f74fb7ad9fb1fe9b642b0c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40445 } } peers { permanent_uuid: "cd13271501f24e7bb8badec00d0aae5d" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 32931 } } peers { permanent_uuid: "58dfca13e8144471a3687f2ca3e8463c" member_type: VOTER last_known_addr { host: "127.20.1.188" port: 46379 } } peers { permanent_uuid: "75c0012122af495a95fffc7a92fc22a4" member_type: VOTER last_known_addr { host: "127.20.1.187" port: 37329 } attrs { promote: false } } }
I20260620 02:02:48.376041 1736 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd13271501f24e7bb8badec00d0aae5d [sys.catalog]: This master's current role is: LEADER
W20260620 02:02:48.884233 1770 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:02:48.884668 1770 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:02:48.884810 1770 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:02:48.897818 1770 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:02:48.898746 1770 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:02:48.915247 1770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:39129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=35649
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379,127.20.1.187:37329
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:02:48.918403 1770 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:02:48.920341 1770 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:02:48.941838 1775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:48.942559 1776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:48.944337 1778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:48.948081 1770 server_base.cc:1061] running on GCE node
I20260620 02:02:48.949060 1770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:02:48.951228 1770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:02:48.952669 1770 hybrid_clock.cc:648] HybridClock initialized: now 1781920968952489 us; error 85 us; skew 500 ppm
I20260620 02:02:48.953812 1770 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:02:48 dist-test-slave-wg14 krb5kdc[561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920968, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:02:48.960191 1770 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:02:48.962457 1770 webserver.cc:533] Webserver started at http://127.20.1.129:35649/ using document root <none> and password file <none>
I20260620 02:02:48.963367 1770 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:48.963505 1770 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:48.974367 1770 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.009s sys 0.000s
I20260620 02:02:48.978624 1785 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:48.981432 1770 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 02:02:48.982327 1770 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "053b7949c7f84aab8620669b9d8d9786"
format_stamp: "Formatted at 2026-06-20 02:02:43 on dist-test-slave-wg14"
I20260620 02:02:48.983597 1770 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:49.006234 1770 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:02:49.013084 1770 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:02:49.013863 1770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:02:49.016824 1770 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:02:49.020830 1770 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:02:49.021163 1770 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:49.021317 1770 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:02:49.021399 1770 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:49.080231 1770 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:39129
I20260620 02:02:49.080302 1899 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:39129 every 8 connection(s)
I20260620 02:02:49.082211 1770 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:02:49.089893 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 1770
Jun 20 02:02:49 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Jun 20 02:02:49 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920968, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.188@KRBTEST.COM
Jun 20 02:02:49 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920968, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
Jun 20 02:02:49 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920968, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:02:49.185802 1901 heartbeater.cc:344] Connected to a master server at 127.20.1.188:46379
I20260620 02:02:49.186311 1901 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:49.187750 1901 heartbeater.cc:507] Master 127.20.1.188:46379 requested a full tablet report, sending...
Jun 20 02:02:49 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920968, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
I20260620 02:02:49.191509 1061 ts_manager.cc:194] Re-registered known tserver with Master: 053b7949c7f84aab8620669b9d8d9786 (127.20.1.129:39129)
I20260620 02:02:49.194923 1900 heartbeater.cc:344] Connected to a master server at 127.20.1.187:37329
I20260620 02:02:49.195340 1900 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:49.196264 1900 heartbeater.cc:507] Master 127.20.1.187:37329 requested a full tablet report, sending...
I20260620 02:02:49.199360 1458 ts_manager.cc:194] Re-registered known tserver with Master: 053b7949c7f84aab8620669b9d8d9786 (127.20.1.129:39129)
I20260620 02:02:49.200299 1902 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40445
I20260620 02:02:49.200551 1902 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:49.201229 1902 heartbeater.cc:507] Master 127.20.1.190:40445 requested a full tablet report, sending...
I20260620 02:02:49.203655 1704 ts_manager.cc:194] Registered new tserver with Master: 053b7949c7f84aab8620669b9d8d9786 (127.20.1.129:39129)
I20260620 02:02:49.225112 1903 heartbeater.cc:344] Connected to a master server at 127.20.1.189:32931
I20260620 02:02:49.226025 1903 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:49.227064 1903 heartbeater.cc:507] Master 127.20.1.189:32931 requested a full tablet report, sending...
I20260620 02:02:49.228938 983 ts_manager.cc:194] Re-registered known tserver with Master: 053b7949c7f84aab8620669b9d8d9786 (127.20.1.129:39129)
I20260620 02:02:49.231112 983 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:45597
I20260620 02:02:49.341178 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 1770
W20260620 02:02:49.342743 1918 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:02:49.338065 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.20.1.129:39129)
0620 02:02:49.338770 (+ 705us) server_negotiation.cc:207] Beginning negotiation
0620 02:02:49.338779 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0620 02:02:49.338817 (+ 38us) server_negotiation.cc:408] Connection header received
0620 02:02:49.338891 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:02:49.338899 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:02:49.339306 (+ 407us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:02:49.339556 (+ 250us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:02:49.340994 (+ 1438us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45138: BlockingRecv error: recv got EOF from 127.0.0.1:45138 (error 108)
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":359,"threads_started":1}
W20260620 02:02:49.344136 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 1770: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:39129: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:02:49.344287 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 1770
W20260620 02:02:49.344419 1919 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:02:49.342283 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:45144 (local address 127.20.1.129:39129)
0620 02:02:49.342962 (+ 679us) server_negotiation.cc:207] Beginning negotiation
0620 02:02:49.342968 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0620 02:02:49.342994 (+ 26us) server_negotiation.cc:408] Connection header received
0620 02:02:49.343070 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:02:49.343076 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:02:49.343194 (+ 118us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:02:49.343321 (+ 127us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:02:49.344140 (+ 819us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45144: BlockingRecv error: recv got EOF from 127.0.0.1:45144 (error 108)
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":358,"threads_started":1}
I20260620 02:02:49.361577 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:39129
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379
--webserver_port=35649
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379,127.20.1.187:37329
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:02:49.871685 1920 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:02:49.873101 1920 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:02:49.873294 1920 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260620 02:02:49.873384 1920 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:02:49.886215 1920 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:02:49.886559 1920 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:02:49.903721 1920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:39129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=35649
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379,127.20.1.187:37329
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:02:49.905577 1920 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:02:49.907305 1920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:02:49.925976 1926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:49.926846 1925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:49.927008 1928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:49.928404 1920 server_base.cc:1061] running on GCE node
I20260620 02:02:49.929234 1920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:02:49.932945 1920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:02:49.934393 1920 hybrid_clock.cc:648] HybridClock initialized: now 1781920969934183 us; error 143 us; skew 500 ppm
I20260620 02:02:49.934808 1920 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:02:49 dist-test-slave-wg14 krb5kdc[561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920969, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:02:49.942943 1920 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:02:49.945952 1920 webserver.cc:533] Webserver started at http://127.20.1.129:35649/ using document root <none> and password file <none>
I20260620 02:02:49.947055 1920 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:49.947266 1920 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:49.956593 1920 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 02:02:49.965279 1935 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:49.967640 1920 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.007s sys 0.002s
I20260620 02:02:49.967906 1920 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "053b7949c7f84aab8620669b9d8d9786"
format_stamp: "Formatted at 2026-06-20 02:02:43 on dist-test-slave-wg14"
I20260620 02:02:49.968837 1920 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:50.004987 1920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:02:50.018019 1920 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:02:50.018452 1920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:02:50.022210 1920 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:02:50.024820 1920 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:02:50.024950 1920 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:50.025079 1920 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:02:50.025135 1920 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 02:02:50.025923 1920 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 02:02:50.026023 1920 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 02:02:50.026141 1920 fault_injection.cc:59] injecting fault for kudu (pid 1920): FLAGS_fault_before_start (process will exit)
I20260620 02:02:50.055768 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:39129
--local_ip_for_outbound_sockets=127.20.1.129
--tserver_master_addrs=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379
--webserver_port=35649
--webserver_interface=127.20.1.129
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379,127.20.1.187:37329 with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:02:50.691493 1943 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:02:50.691902 1943 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:02:50.692029 1943 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:02:50.718392 1943 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:02:50.719556 1943 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:02:50.736229 1943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38495,127.20.1.149:40793,127.20.1.150:46291,127.20.1.151:36863,127.20.1.152:45749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:39129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=35649
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:40445,127.20.1.189:32931,127.20.1.188:46379,127.20.1.187:37329
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:02:50.738039 1943 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:02:50.741570 1943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:02:50.761363 1949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:50.761929 1948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:50.762506 1951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:50.764485 1943 server_base.cc:1061] running on GCE node
I20260620 02:02:50.765987 1943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:02:50.768524 1943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:02:50.772655 1943 hybrid_clock.cc:648] HybridClock initialized: now 1781920970772376 us; error 99 us; skew 500 ppm
I20260620 02:02:50.773425 1943 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:02:50 dist-test-slave-wg14 krb5kdc[561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920970, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:02:50.777869 1943 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:02:50.781939 1943 webserver.cc:533] Webserver started at http://127.20.1.129:35649/ using document root <none> and password file <none>
I20260620 02:02:50.782994 1943 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:50.783198 1943 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:50.791997 1943 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 02:02:50.797262 1958 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:50.800349 1943 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:02:50.800613 1943 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "053b7949c7f84aab8620669b9d8d9786"
format_stamp: "Formatted at 2026-06-20 02:02:43 on dist-test-slave-wg14"
I20260620 02:02:50.801715 1943 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:50.820042 1943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:02:50.825788 1943 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:02:50.826211 1943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:02:50.827857 1943 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:02:50.830161 1943 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:02:50.830323 1943 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:50.830449 1943 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:02:50.830530 1943 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:50.894085 1943 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:39129
I20260620 02:02:50.894129 2071 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:39129 every 8 connection(s)
I20260620 02:02:50.896320 1943 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:02:50.909854 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 1943
I20260620 02:02:50.910670 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 1943
Jun 20 02:02:50 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920970, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.187@KRBTEST.COM
Jun 20 02:02:50 dist-test-slave-wg14 krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920970, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.189@KRBTEST.COM
W20260620 02:02:50.960256 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 1943: Not authorized: Client connection negotiation failed: client connection to 127.20.1.129:39129: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:02:50.961621 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 1943
I20260620 02:02:50.993747 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 1668
W20260620 02:02:50.994881 2085 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:02:50.934405 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:58221 (local address 127.20.1.190:40445)
0620 02:02:50.936211 (+ 1806us) server_negotiation.cc:207] Beginning negotiation
0620 02:02:50.936219 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 02:02:50.953112 (+ 16893us) server_negotiation.cc:408] Connection header received
0620 02:02:50.953442 (+ 330us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:02:50.953448 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:02:50.953732 (+ 284us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:02:50.953828 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:02:50.958402 (+ 4574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 02:02:50.965286 (+ 6884us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:02:50.993374 (+ 28088us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:58221: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":1669,"thread_start_us":1564,"threads_started":1}
W20260620 02:02:50.995213 2080 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:02:50.913620 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:47069 (local address 127.20.1.189:32931)
0620 02:02:50.933232 (+ 19612us) server_negotiation.cc:207] Beginning negotiation
0620 02:02:50.933240 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 02:02:50.933270 (+ 30us) server_negotiation.cc:408] Connection header received
0620 02:02:50.933360 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:02:50.933367 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:02:50.933460 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:02:50.933972 (+ 512us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:02:50.938229 (+ 4257us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 02:02:50.940542 (+ 2313us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:02:50.947849 (+ 7307us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 02:02:50.948578 (+ 729us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0620 02:02:50.993923 (+ 45345us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:47069: BlockingRecv error: recv got EOF from 127.20.1.129:47069 (error 108)
Metrics: {"server-negotiator.queue_time_us":19341,"thread_start_us":19201,"threads_started":1}
W20260620 02:02:50.995250 2083 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:02:50.917909 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:37081 (local address 127.20.1.188:46379)
0620 02:02:50.923978 (+ 6069us) server_negotiation.cc:207] Beginning negotiation
0620 02:02:50.923986 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0620 02:02:50.936477 (+ 12491us) server_negotiation.cc:408] Connection header received
0620 02:02:50.938843 (+ 2366us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:02:50.938851 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:02:50.938956 (+ 105us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:02:50.939694 (+ 738us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:02:50.957077 (+ 17383us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 02:02:50.960257 (+ 3180us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:02:50.993874 (+ 33617us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:37081: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":5598,"thread_start_us":5194,"threads_started":1}
W20260620 02:02:50.996973 2077 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:02:50.912003 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.1.129:38991 (local address 127.20.1.187:37329)
0620 02:02:50.928289 (+ 16286us) server_negotiation.cc:207] Beginning negotiation
0620 02:02:50.928296 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 02:02:50.928328 (+ 32us) server_negotiation.cc:408] Connection header received
0620 02:02:50.928426 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:02:50.928431 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:02:50.928550 (+ 119us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:02:50.928726 (+ 176us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:02:50.934849 (+ 6123us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 02:02:50.937413 (+ 2564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:02:50.944821 (+ 7408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0620 02:02:50.945551 (+ 730us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0620 02:02:50.960259 (+ 14708us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0620 02:02:50.960316 (+ 57us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0620 02:02:50.960321 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0620 02:02:50.960384 (+ 63us) server_negotiation.cc:923] calling sasl_server_start()
0620 02:02:50.966377 (+ 5993us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0620 02:02:50.996025 (+ 29648us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.1.129:38991: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":16121,"thread_start_us":15995,"threads_started":1}
W20260620 02:02:50.997972 20486 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 1668: Not authorized: Client connection negotiation failed: client connection to 127.20.1.190:40445: server requires authentication, but client does not have Kerberos credentials available
W20260620 02:02:50.998153 2085 negotiation.cc:336] Failed RPC negotiation. Trace:
0620 02:02:50.996006 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:35318 (local address 127.20.1.190:40445)
0620 02:02:50.996188 (+ 182us) server_negotiation.cc:207] Beginning negotiation
0620 02:02:50.996195 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0620 02:02:50.996376 (+ 181us) server_negotiation.cc:408] Connection header received
0620 02:02:50.996644 (+ 268us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0620 02:02:50.996650 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0620 02:02:50.996729 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0620 02:02:50.996828 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0620 02:02:50.997864 (+ 1036us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35318: BlockingRecv error: recv got EOF from 127.0.0.1:35318 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:02:50.998175 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 1668
I20260620 02:02:51.025208 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 950
I20260620 02:02:51.027058 1013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:02:51.197489 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 950
I20260620 02:02:51.226742 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 1028
I20260620 02:02:51.228533 1092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:02:51.381687 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 1028
I20260620 02:02:51.408481 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 1313
I20260620 02:02:51.410953 1489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:02:51.571851 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 1313
Shutting down hive metastore.
2026-06-20T02:02:52Z chronyd exiting
2026-06-20T02:02:52Z chronyd exiting
2026-06-20T02:02:52Z chronyd exiting
2026-06-20T02:02:52Z chronyd exiting
2026-06-20T02:02:52Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/11 (46657 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/0
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:02:52 dist-test-slave-wg14 krb5kdc[2098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:02:52 dist-test-slave-wg14 krb5kdc[2098](info): set up 2 sockets
Jun 20 02:02:52 dist-test-slave-wg14 krb5kdc[2098](info): commencing operation
krb5kdc: starting...
W20260620 02:02:55.005546 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.883s user 0.003s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 02:02:55 dist-test-slave-wg14 krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:02:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:02:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:02:55.623430 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40581
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:40581
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:02:56.339710 2114 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:02:56.341080 2114 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:02:56.342020 2114 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:02:56.357197 2114 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:02:56.357357 2114 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:02:56.357443 2114 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:02:56.357517 2114 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:02:56.377177 2114 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:40581
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:40581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:02:56.379762 2114 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:02:56.382408 2114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:02:56.400759 2119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:56.400779 2120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:56.401062 2122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:56.404150 2114 server_base.cc:1061] running on GCE node
I20260620 02:02:56.405541 2114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:02:56.408962 2114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:02:56.413015 2114 hybrid_clock.cc:648] HybridClock initialized: now 1781920976412803 us; error 144 us; skew 500 ppm
I20260620 02:02:56.413697 2114 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:02:56 dist-test-slave-wg14 krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920976, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:02:56.419262 2114 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:02:56.423763 2114 webserver.cc:533] Webserver started at http://127.20.1.190:40919/ using document root <none> and password file <none>
I20260620 02:02:56.425343 2114 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:56.425668 2114 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:56.426138 2114 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:02:56.429409 2114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "aad303283c0c4af593151e54e24d3118"
format_stamp: "Formatted at 2026-06-20 02:02:56 on dist-test-slave-wg14"
I20260620 02:02:56.430955 2114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "aad303283c0c4af593151e54e24d3118"
format_stamp: "Formatted at 2026-06-20 02:02:56 on dist-test-slave-wg14"
I20260620 02:02:56.441552 2114 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.007s sys 0.001s
I20260620 02:02:56.451650 2129 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:56.453950 2114 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260620 02:02:56.454480 2114 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "aad303283c0c4af593151e54e24d3118"
format_stamp: "Formatted at 2026-06-20 02:02:56 on dist-test-slave-wg14"
I20260620 02:02:56.454991 2114 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:56.489367 2114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:02:56.500972 2114 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:02:56.501374 2114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:02:56.551564 2181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:40581 every 8 connection(s)
I20260620 02:02:56.551564 2114 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:40581
I20260620 02:02:56.553975 2114 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:02:56.561635 2182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:02:56.562865 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 2114
I20260620 02:02:56.563938 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:02:56.580693 2182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118: Bootstrap starting.
I20260620 02:02:56.585872 2182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118: Neither blocks nor log segments found. Creating new log.
I20260620 02:02:56.587422 2182 log.cc:826] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:02:56 dist-test-slave-wg14 krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:02:56.594585 2182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118: No bootstrap required, opened a new log
I20260620 02:02:56.603650 2182 raft_consensus.cc:359] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad303283c0c4af593151e54e24d3118" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40581 } }
I20260620 02:02:56.604820 2182 raft_consensus.cc:385] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:02:56.605098 2182 raft_consensus.cc:740] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad303283c0c4af593151e54e24d3118, State: Initialized, Role: FOLLOWER
I20260620 02:02:56.606071 2182 consensus_queue.cc:260] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad303283c0c4af593151e54e24d3118" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40581 } }
I20260620 02:02:56.606383 2182 raft_consensus.cc:399] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:02:56.606623 2182 raft_consensus.cc:493] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:02:56.607164 2182 raft_consensus.cc:3060] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:02:56.614017 2182 raft_consensus.cc:515] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad303283c0c4af593151e54e24d3118" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40581 } }
I20260620 02:02:56.614894 2182 leader_election.cc:304] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aad303283c0c4af593151e54e24d3118; no voters:
I20260620 02:02:56.616074 2182 leader_election.cc:290] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:02:56.616696 2187 raft_consensus.cc:2804] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:02:56.617965 2187 raft_consensus.cc:697] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [term 1 LEADER]: Becoming Leader. State: Replica: aad303283c0c4af593151e54e24d3118, State: Running, Role: LEADER
I20260620 02:02:56.618817 2187 consensus_queue.cc:237] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad303283c0c4af593151e54e24d3118" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40581 } }
I20260620 02:02:56.620867 2182 sys_catalog.cc:565] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:02:56.626204 2188 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad303283c0c4af593151e54e24d3118. Latest consensus state: current_term: 1 leader_uuid: "aad303283c0c4af593151e54e24d3118" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad303283c0c4af593151e54e24d3118" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40581 } } }
I20260620 02:02:56.627362 2188 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [sys.catalog]: This master's current role is: LEADER
I20260620 02:02:56.631791 2189 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aad303283c0c4af593151e54e24d3118" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad303283c0c4af593151e54e24d3118" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 40581 } } }
I20260620 02:02:56.635217 2189 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118 [sys.catalog]: This master's current role is: LEADER
I20260620 02:02:56.640990 2194 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:02:56.655752 2194 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:02:56.671479 2194 catalog_manager.cc:1382] Generated new cluster ID: 2891c80824034ded8f66aec4a2edec74
I20260620 02:02:56.671649 2194 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:02:56.709800 2194 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:02:56.717278 2194 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:02:56.745146 2194 catalog_manager.cc:6069] T 00000000000000000000000000000000 P aad303283c0c4af593151e54e24d3118: Generated new TSK 0
I20260620 02:02:56.749742 2194 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:02:56.884159 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.121s user 0.004s sys 0.044s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:02:56.998107 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.114s user 0.000s sys 0.037s
I20260620 02:02:56.999482 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:40581
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:02:57.642932 2210 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:02:57.643373 2210 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:02:57.643488 2210 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:02:57.655876 2210 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:02:57.656527 2210 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:02:57.670240 2210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:40581
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:02:57.673152 2210 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:02:57.676425 2210 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:02:57.695875 2218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:57.695897 2216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:02:57.695884 2215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:02:57.698308 2210 server_base.cc:1061] running on GCE node
I20260620 02:02:57.699115 2210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:02:57.700850 2210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:02:57.702590 2210 hybrid_clock.cc:648] HybridClock initialized: now 1781920977702450 us; error 138 us; skew 500 ppm
I20260620 02:02:57.703085 2210 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:02:57 dist-test-slave-wg14 krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920977, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:02:57.707775 2210 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:02:57.710920 2210 webserver.cc:533] Webserver started at http://127.20.1.129:34465/ using document root <none> and password file <none>
I20260620 02:02:57.713372 2210 fs_manager.cc:362] Metadata directory not provided
I20260620 02:02:57.713603 2210 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:02:57.714111 2210 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:02:57.717792 2210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "b34d45816e814b25a4fabf6dc6955a54"
format_stamp: "Formatted at 2026-06-20 02:02:57 on dist-test-slave-wg14"
I20260620 02:02:57.719501 2210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "b34d45816e814b25a4fabf6dc6955a54"
format_stamp: "Formatted at 2026-06-20 02:02:57 on dist-test-slave-wg14"
I20260620 02:02:57.728971 2210 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260620 02:02:57.735518 2225 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:57.739928 2210 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260620 02:02:57.740463 2210 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "b34d45816e814b25a4fabf6dc6955a54"
format_stamp: "Formatted at 2026-06-20 02:02:57 on dist-test-slave-wg14"
I20260620 02:02:57.740825 2210 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:02:57.768801 2210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:02:57.776105 2210 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:02:57.776546 2210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:02:57.778776 2210 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:02:57.782585 2210 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:02:57.782716 2210 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:02:57.782855 2210 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:02:57.783288 2210 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.002s sys 0.000s
I20260620 02:02:57.864251 2210 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:45145
I20260620 02:02:57.864468 2338 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:45145 every 8 connection(s)
I20260620 02:02:57.866521 2210 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:02:57.867558 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 2210
I20260620 02:02:57.870507 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 02:02:57 dist-test-slave-wg14 krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920977, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:02:57.913132 2339 heartbeater.cc:344] Connected to a master server at 127.20.1.190:40581
I20260620 02:02:57.913650 2339 heartbeater.cc:461] Registering TS with master...
I20260620 02:02:57.914799 2339 heartbeater.cc:507] Master 127.20.1.190:40581 requested a full tablet report, sending...
I20260620 02:02:57.918880 2146 ts_manager.cc:194] Registered new tserver with Master: b34d45816e814b25a4fabf6dc6955a54 (127.20.1.129:45145)
I20260620 02:02:57.922271 2146 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:39509
I20260620 02:02:57.923087 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Jun 20 02:02:57 dist-test-slave-wg14 krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:02:58 dist-test-slave-wg14 krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.20.1.190@KRBTEST.COM
I20260620 02:02:58.012473 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 2210
Jun 20 02:02:58 dist-test-slave-wg14 krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:02:58.043433 2334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:02:58.173205 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 2210
I20260620 02:02:58.195580 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 2114
I20260620 02:02:58.198366 2176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:02:58.398123 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 2114
2026-06-20T02:02:58Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/0 (6342 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/1
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:02:58 dist-test-slave-wg14 krb5kdc[2352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:02:58 dist-test-slave-wg14 krb5kdc[2352](info): set up 2 sockets
Jun 20 02:02:58 dist-test-slave-wg14 krb5kdc[2352](info): commencing operation
krb5kdc: starting...
W20260620 02:03:01.503866 20486 mini_kdc.cc:121] Time spent starting KDC: real 3.042s user 0.001s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 02:03:01 dist-test-slave-wg14 krb5kdc[2352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:03:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:03:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:02.025816 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.190: real 0.126s user 0.002s sys 0.038s
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:02.145785 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.190: real 0.119s user 0.002s sys 0.025s
I20260620 02:03:02.149394 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:46023
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:41629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:46023
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:02.882167 2368 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:02.882810 2368 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:02.882964 2368 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:02.894591 2368 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:03:02.894760 2368 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:02.894840 2368 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:03:02.894917 2368 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:03:02.915966 2368 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:46023
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:46023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:02.922191 2368 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:02.924088 2368 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:02.937840 2374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:02.938609 2373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:02.939673 2376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:02.941308 2368 server_base.cc:1061] running on GCE node
I20260620 02:03:02.942580 2368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:02.945072 2368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:02.946796 2368 hybrid_clock.cc:648] HybridClock initialized: now 1781920982946716 us; error 67 us; skew 500 ppm
I20260620 02:03:02.947417 2368 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:02 dist-test-slave-wg14 krb5kdc[2352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920982, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:02.954777 2368 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:03:02.959270 2368 webserver.cc:533] Webserver started at http://127.20.1.190:33335/ using document root <none> and password file <none>
I20260620 02:03:02.960268 2368 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:02.960395 2368 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:02.960925 2368 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:02.965553 2368 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "01b1ab5b27544785aa293470f23cb4d4"
format_stamp: "Formatted at 2026-06-20 02:03:02 on dist-test-slave-wg14"
I20260620 02:03:02.966876 2368 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "01b1ab5b27544785aa293470f23cb4d4"
format_stamp: "Formatted at 2026-06-20 02:03:02 on dist-test-slave-wg14"
I20260620 02:03:02.980015 2368 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.010s sys 0.001s
I20260620 02:03:02.987591 2383 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:02.991432 2368 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.005s sys 0.000s
I20260620 02:03:02.991837 2368 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "01b1ab5b27544785aa293470f23cb4d4"
format_stamp: "Formatted at 2026-06-20 02:03:02 on dist-test-slave-wg14"
I20260620 02:03:02.992161 2368 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:03.038482 2368 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:03.055253 2368 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:03.055753 2368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:03.127405 2368 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:46023
I20260620 02:03:03.127405 2434 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:46023 every 8 connection(s)
I20260620 02:03:03.129927 2368 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:03:03.131017 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 2368
I20260620 02:03:03.131248 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:03:03.138463 2435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:03:03 dist-test-slave-wg14 krb5kdc[2352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:03.164081 2435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4: Bootstrap starting.
I20260620 02:03:03.175741 2435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4: Neither blocks nor log segments found. Creating new log.
I20260620 02:03:03.177172 2435 log.cc:826] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4: Log is configured to *not* fsync() on all Append() calls
I20260620 02:03:03.185322 2435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4: No bootstrap required, opened a new log
I20260620 02:03:03.197479 2435 raft_consensus.cc:359] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b1ab5b27544785aa293470f23cb4d4" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46023 } }
I20260620 02:03:03.198005 2435 raft_consensus.cc:385] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:03:03.198220 2435 raft_consensus.cc:740] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b1ab5b27544785aa293470f23cb4d4, State: Initialized, Role: FOLLOWER
I20260620 02:03:03.199097 2435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b1ab5b27544785aa293470f23cb4d4" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46023 } }
I20260620 02:03:03.199496 2435 raft_consensus.cc:399] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:03:03.199662 2435 raft_consensus.cc:493] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:03:03.199842 2435 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:03:03.215395 2435 raft_consensus.cc:515] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b1ab5b27544785aa293470f23cb4d4" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46023 } }
I20260620 02:03:03.216892 2435 leader_election.cc:304] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01b1ab5b27544785aa293470f23cb4d4; no voters:
I20260620 02:03:03.218091 2435 leader_election.cc:290] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:03:03.218303 2440 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:03:03.226246 2440 raft_consensus.cc:697] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [term 1 LEADER]: Becoming Leader. State: Replica: 01b1ab5b27544785aa293470f23cb4d4, State: Running, Role: LEADER
I20260620 02:03:03.227340 2440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b1ab5b27544785aa293470f23cb4d4" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46023 } }
I20260620 02:03:03.228449 2435 sys_catalog.cc:565] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:03:03.232347 2442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01b1ab5b27544785aa293470f23cb4d4. Latest consensus state: current_term: 1 leader_uuid: "01b1ab5b27544785aa293470f23cb4d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b1ab5b27544785aa293470f23cb4d4" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46023 } } }
I20260620 02:03:03.232923 2442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:03.237803 2441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01b1ab5b27544785aa293470f23cb4d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b1ab5b27544785aa293470f23cb4d4" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46023 } } }
I20260620 02:03:03.239061 2441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:03.241671 2445 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:03:03.278359 2445 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:03:03.332420 2445 catalog_manager.cc:1382] Generated new cluster ID: 7a7baca25a354fe39f452b6c598d22d7
I20260620 02:03:03.332588 2445 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:03:03.373756 2445 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:03:03.376044 2445 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:03:03.434510 2445 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 01b1ab5b27544785aa293470f23cb4d4: Generated new TSK 0
I20260620 02:03:03.435972 2445 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:03.585270 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.129s user 0.002s sys 0.032s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:03.719456 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.134s user 0.001s sys 0.031s
I20260620 02:03:03.720329 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:46023
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:41629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:04.496430 2463 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:04.497473 2463 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:04.497678 2463 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:04.517117 2463 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:04.517406 2463 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:03:04.539697 2463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:41629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:46023
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:04.546355 2463 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:04.548183 2463 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:04.576160 2469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:04.576239 2468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:04.577410 2471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:04.578248 2463 server_base.cc:1061] running on GCE node
I20260620 02:03:04.579026 2463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:04.580578 2463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:04.594007 2463 hybrid_clock.cc:648] HybridClock initialized: now 1781920984593345 us; error 604 us; skew 500 ppm
I20260620 02:03:04.594705 2463 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:04 dist-test-slave-wg14 krb5kdc[2352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920984, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:04.599315 2463 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:03:04.602730 2463 webserver.cc:533] Webserver started at http://127.20.1.129:42401/ using document root <none> and password file <none>
I20260620 02:03:04.603933 2463 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:04.604197 2463 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:04.604707 2463 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:04.608901 2463 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "b8409b997e0a4f01acbf0603b96fcee1"
format_stamp: "Formatted at 2026-06-20 02:03:04 on dist-test-slave-wg14"
I20260620 02:03:04.611177 2463 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "b8409b997e0a4f01acbf0603b96fcee1"
format_stamp: "Formatted at 2026-06-20 02:03:04 on dist-test-slave-wg14"
I20260620 02:03:04.620440 2463 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260620 02:03:04.626331 2478 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:04.628873 2463 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260620 02:03:04.629243 2463 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "b8409b997e0a4f01acbf0603b96fcee1"
format_stamp: "Formatted at 2026-06-20 02:03:04 on dist-test-slave-wg14"
I20260620 02:03:04.629540 2463 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:04.661275 2463 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:04.667917 2463 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:04.668378 2463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:04.669937 2463 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:03:04.674902 2463 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:03:04.675071 2463 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:04.675321 2463 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:03:04.675391 2463 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:04.756644 2463 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:34127
I20260620 02:03:04.756748 2592 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:34127 every 8 connection(s)
I20260620 02:03:04.758677 2463 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:03:04.767115 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 2463
I20260620 02:03:04.767306 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 02:03:04 dist-test-slave-wg14 krb5kdc[2352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920984, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:04.805060 2593 heartbeater.cc:344] Connected to a master server at 127.20.1.190:46023
I20260620 02:03:04.805560 2593 heartbeater.cc:461] Registering TS with master...
I20260620 02:03:04.811650 2593 heartbeater.cc:507] Master 127.20.1.190:46023 requested a full tablet report, sending...
I20260620 02:03:04.815698 2400 ts_manager.cc:194] Registered new tserver with Master: b8409b997e0a4f01acbf0603b96fcee1 (127.20.1.129:34127)
I20260620 02:03:04.818056 2400 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:60257
I20260620 02:03:04.820225 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Jun 20 02:03:04 dist-test-slave-wg14 krb5kdc[2352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:03:04 dist-test-slave-wg14 krb5kdc[2352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.20.1.190@KRBTEST.COM
I20260620 02:03:04.916755 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 2463
Jun 20 02:03:04 dist-test-slave-wg14 krb5kdc[2352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:03:04.954368 2588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:05.116176 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 2463
I20260620 02:03:05.147823 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 2368
I20260620 02:03:05.151321 2430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:05.356873 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 2368
2026-06-20T02:03:05Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/1 (6962 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/2
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:03:05 dist-test-slave-wg14 krb5kdc[2606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:03:05 dist-test-slave-wg14 krb5kdc[2606](info): set up 2 sockets
Jun 20 02:03:05 dist-test-slave-wg14 krb5kdc[2606](info): commencing operation
krb5kdc: starting...
W20260620 02:03:08.313740 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.890s user 0.001s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 02:03:08 dist-test-slave-wg14 krb5kdc[2606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:03:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:03:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:03:08.888321 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:41777
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:45099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:41777
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:09.693727 2622 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:09.694708 2622 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:09.694856 2622 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:09.715103 2622 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:03:09.715320 2622 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:09.715389 2622 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:03:09.715443 2622 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:03:09.737454 2622 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:41777
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:41777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:09.739521 2622 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:09.741122 2622 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:09.758827 2628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:09.758828 2627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:09.759384 2630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:09.760267 2622 server_base.cc:1061] running on GCE node
I20260620 02:03:09.764616 2622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:09.769102 2622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:09.772905 2622 hybrid_clock.cc:648] HybridClock initialized: now 1781920989772747 us; error 134 us; skew 500 ppm
I20260620 02:03:09.773960 2622 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:09 dist-test-slave-wg14 krb5kdc[2606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920989, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:09.778949 2622 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:03:09.785678 2622 webserver.cc:533] Webserver started at http://127.20.1.190:45141/ using document root <none> and password file <none>
I20260620 02:03:09.788443 2622 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:09.790652 2622 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:09.791522 2622 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:09.796355 2622 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "08250ed9f0104d26a6b597a243ac2dbc"
format_stamp: "Formatted at 2026-06-20 02:03:09 on dist-test-slave-wg14"
I20260620 02:03:09.797847 2622 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "08250ed9f0104d26a6b597a243ac2dbc"
format_stamp: "Formatted at 2026-06-20 02:03:09 on dist-test-slave-wg14"
I20260620 02:03:09.813807 2622 fs_manager.cc:696] Time spent creating directory manager: real 0.015s user 0.017s sys 0.000s
I20260620 02:03:09.819294 2637 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:09.823036 2622 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.001s sys 0.002s
I20260620 02:03:09.823375 2622 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "08250ed9f0104d26a6b597a243ac2dbc"
format_stamp: "Formatted at 2026-06-20 02:03:09 on dist-test-slave-wg14"
I20260620 02:03:09.824137 2622 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:09.866400 2622 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:09.880901 2622 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:09.882141 2622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:09.926987 2689 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:41777 every 8 connection(s)
I20260620 02:03:09.927426 2622 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:41777
I20260620 02:03:09.933920 2622 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:03:09.936736 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 2622
I20260620 02:03:09.936928 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:03:09.944672 2690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:03:09 dist-test-slave-wg14 krb5kdc[2606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:09.963945 2690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc: Bootstrap starting.
I20260620 02:03:09.970181 2690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc: Neither blocks nor log segments found. Creating new log.
I20260620 02:03:09.971768 2690 log.cc:826] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc: Log is configured to *not* fsync() on all Append() calls
I20260620 02:03:09.978916 2690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc: No bootstrap required, opened a new log
I20260620 02:03:09.990541 2690 raft_consensus.cc:359] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08250ed9f0104d26a6b597a243ac2dbc" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41777 } }
I20260620 02:03:09.991104 2690 raft_consensus.cc:385] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:03:09.991335 2690 raft_consensus.cc:740] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08250ed9f0104d26a6b597a243ac2dbc, State: Initialized, Role: FOLLOWER
I20260620 02:03:09.992333 2690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08250ed9f0104d26a6b597a243ac2dbc" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41777 } }
I20260620 02:03:09.992682 2690 raft_consensus.cc:399] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:03:09.992856 2690 raft_consensus.cc:493] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:03:09.993271 2690 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:03:09.997963 2690 raft_consensus.cc:515] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08250ed9f0104d26a6b597a243ac2dbc" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41777 } }
I20260620 02:03:09.999488 2690 leader_election.cc:304] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08250ed9f0104d26a6b597a243ac2dbc; no voters:
I20260620 02:03:10.000449 2690 leader_election.cc:290] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:03:10.000662 2695 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:03:10.004659 2695 raft_consensus.cc:697] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [term 1 LEADER]: Becoming Leader. State: Replica: 08250ed9f0104d26a6b597a243ac2dbc, State: Running, Role: LEADER
I20260620 02:03:10.004659 2690 sys_catalog.cc:565] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:03:10.007305 2695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08250ed9f0104d26a6b597a243ac2dbc" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41777 } }
I20260620 02:03:10.013723 2697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08250ed9f0104d26a6b597a243ac2dbc. Latest consensus state: current_term: 1 leader_uuid: "08250ed9f0104d26a6b597a243ac2dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08250ed9f0104d26a6b597a243ac2dbc" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41777 } } }
I20260620 02:03:10.013723 2696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08250ed9f0104d26a6b597a243ac2dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08250ed9f0104d26a6b597a243ac2dbc" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 41777 } } }
I20260620 02:03:10.016141 2697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:10.016536 2696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:10.018616 2705 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:03:10.038440 2705 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:03:10.068723 2705 catalog_manager.cc:1382] Generated new cluster ID: 46eef8040fe44a23be628b6f1cb1fe28
I20260620 02:03:10.068920 2705 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:03:10.137094 2705 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:03:10.138695 2705 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:03:10.172829 2705 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 08250ed9f0104d26a6b597a243ac2dbc: Generated new TSK 0
I20260620 02:03:10.174094 2705 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:10.343359 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.143s user 0.003s sys 0.026s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:10.463310 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.120s user 0.002s sys 0.043s
I20260620 02:03:10.464921 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:41777
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:45099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:11.231748 2718 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:11.232332 2718 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:11.232520 2718 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:11.245112 2718 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:11.245587 2718 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:03:11.261974 2718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:45099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:41777
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:11.264184 2718 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:11.265980 2718 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:11.278649 2724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:11.278441 2723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:11.280692 2726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:11.282641 2718 server_base.cc:1061] running on GCE node
I20260620 02:03:11.283892 2718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:11.285415 2718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:11.288962 2718 hybrid_clock.cc:648] HybridClock initialized: now 1781920991288624 us; error 256 us; skew 500 ppm
I20260620 02:03:11.289743 2718 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:11 dist-test-slave-wg14 krb5kdc[2606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920991, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:11.298611 2718 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:03:11.304744 2718 webserver.cc:533] Webserver started at http://127.20.1.129:35341/ using document root <none> and password file <none>
I20260620 02:03:11.306200 2718 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:11.306393 2718 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:11.306831 2718 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:11.310688 2718 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "94e6e766c8f44aa7bfbf65dea75655b9"
format_stamp: "Formatted at 2026-06-20 02:03:11 on dist-test-slave-wg14"
I20260620 02:03:11.313648 2718 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "94e6e766c8f44aa7bfbf65dea75655b9"
format_stamp: "Formatted at 2026-06-20 02:03:11 on dist-test-slave-wg14"
I20260620 02:03:11.328925 2718 fs_manager.cc:696] Time spent creating directory manager: real 0.013s user 0.010s sys 0.000s
I20260620 02:03:11.338867 2733 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:11.342154 2718 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260620 02:03:11.342445 2718 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "94e6e766c8f44aa7bfbf65dea75655b9"
format_stamp: "Formatted at 2026-06-20 02:03:11 on dist-test-slave-wg14"
I20260620 02:03:11.342749 2718 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:11.398034 2718 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:11.405385 2718 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:11.406512 2718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:11.408320 2718 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:03:11.411580 2718 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:03:11.411751 2718 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:11.411916 2718 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:03:11.411983 2718 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:11.510808 2718 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:36493
I20260620 02:03:11.510814 2848 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:36493 every 8 connection(s)
I20260620 02:03:11.513650 2718 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:03:11.519387 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 2718
I20260620 02:03:11.519717 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 02:03:11 dist-test-slave-wg14 krb5kdc[2606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920991, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:11.556975 2849 heartbeater.cc:344] Connected to a master server at 127.20.1.190:41777
I20260620 02:03:11.557587 2849 heartbeater.cc:461] Registering TS with master...
I20260620 02:03:11.559562 2849 heartbeater.cc:507] Master 127.20.1.190:41777 requested a full tablet report, sending...
I20260620 02:03:11.569268 2654 ts_manager.cc:194] Registered new tserver with Master: 94e6e766c8f44aa7bfbf65dea75655b9 (127.20.1.129:36493)
I20260620 02:03:11.572446 2654 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:39475
I20260620 02:03:11.574533 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Jun 20 02:03:11 dist-test-slave-wg14 krb5kdc[2606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:03:11 dist-test-slave-wg14 krb5kdc[2606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.20.1.190@KRBTEST.COM
I20260620 02:03:11.658825 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 2718
Jun 20 02:03:11 dist-test-slave-wg14 krb5kdc[2606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:03:11.698441 2844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:11.897653 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 2718
I20260620 02:03:11.916738 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 2622
I20260620 02:03:11.919737 2685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:12.081566 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 2622
2026-06-20T02:03:12Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/2 (6717 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/3
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:03:12 dist-test-slave-wg14 krb5kdc[2862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:03:12 dist-test-slave-wg14 krb5kdc[2862](info): set up 2 sockets
Jun 20 02:03:12 dist-test-slave-wg14 krb5kdc[2862](info): commencing operation
krb5kdc: starting...
W20260620 02:03:14.880194 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.740s user 0.004s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 02:03:15 dist-test-slave-wg14 krb5kdc[2862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260620 02:03:15.213362 20486 mini_kdc.cc:305] Time spent kinit for test-admin: real 0.112s user 0.000s sys 0.030s
2026-06-20T02:03:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:03:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:15.485816 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.190: real 0.106s user 0.001s sys 0.019s
I20260620 02:03:15.486829 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:34569
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:34569
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:16.197551 2878 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:16.198467 2878 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:16.198576 2878 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:16.211066 2878 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:03:16.211578 2878 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:16.211742 2878 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:03:16.211818 2878 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:03:16.239925 2878 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:34569
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:34569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:16.247287 2878 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:16.249306 2878 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:16.263469 2884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:16.264456 2883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:16.265219 2886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:16.271989 2878 server_base.cc:1061] running on GCE node
I20260620 02:03:16.273849 2878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:16.277220 2878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:16.278713 2878 hybrid_clock.cc:648] HybridClock initialized: now 1781920996278631 us; error 74 us; skew 500 ppm
I20260620 02:03:16.279328 2878 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:16 dist-test-slave-wg14 krb5kdc[2862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:16.285383 2878 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:03:16.294750 2878 webserver.cc:533] Webserver started at http://127.20.1.190:43331/ using document root <none> and password file <none>
I20260620 02:03:16.297019 2878 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:16.297227 2878 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:16.297725 2878 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:16.304643 2878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "e04f25be9b1a4e81b053316a02d00790"
format_stamp: "Formatted at 2026-06-20 02:03:16 on dist-test-slave-wg14"
I20260620 02:03:16.306277 2878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "e04f25be9b1a4e81b053316a02d00790"
format_stamp: "Formatted at 2026-06-20 02:03:16 on dist-test-slave-wg14"
I20260620 02:03:16.315284 2878 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.011s sys 0.000s
I20260620 02:03:16.322257 2893 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:16.324653 2878 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:03:16.324967 2878 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "e04f25be9b1a4e81b053316a02d00790"
format_stamp: "Formatted at 2026-06-20 02:03:16 on dist-test-slave-wg14"
I20260620 02:03:16.325295 2878 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:16.357057 2878 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:16.371876 2878 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:16.372433 2878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:16.417029 2878 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:34569
I20260620 02:03:16.417059 2945 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:34569 every 8 connection(s)
I20260620 02:03:16.419106 2878 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:03:16.422603 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 2878
I20260620 02:03:16.423274 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:03:16.426895 2946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:03:16.444197 2946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790: Bootstrap starting.
Jun 20 02:03:16 dist-test-slave-wg14 krb5kdc[2862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:16.449903 2946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790: Neither blocks nor log segments found. Creating new log.
I20260620 02:03:16.452759 2946 log.cc:826] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790: Log is configured to *not* fsync() on all Append() calls
I20260620 02:03:16.457633 2946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790: No bootstrap required, opened a new log
I20260620 02:03:16.467554 2946 raft_consensus.cc:359] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04f25be9b1a4e81b053316a02d00790" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34569 } }
I20260620 02:03:16.468473 2946 raft_consensus.cc:385] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:03:16.468638 2946 raft_consensus.cc:740] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04f25be9b1a4e81b053316a02d00790, State: Initialized, Role: FOLLOWER
I20260620 02:03:16.469722 2946 consensus_queue.cc:260] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04f25be9b1a4e81b053316a02d00790" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34569 } }
I20260620 02:03:16.470223 2946 raft_consensus.cc:399] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:03:16.470638 2946 raft_consensus.cc:493] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:03:16.471387 2946 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:03:16.476187 2946 raft_consensus.cc:515] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04f25be9b1a4e81b053316a02d00790" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34569 } }
I20260620 02:03:16.477451 2946 leader_election.cc:304] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e04f25be9b1a4e81b053316a02d00790; no voters:
I20260620 02:03:16.479700 2946 leader_election.cc:290] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:03:16.479887 2951 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:03:16.481523 2951 raft_consensus.cc:697] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [term 1 LEADER]: Becoming Leader. State: Replica: e04f25be9b1a4e81b053316a02d00790, State: Running, Role: LEADER
I20260620 02:03:16.487540 2946 sys_catalog.cc:565] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:03:16.487634 2951 consensus_queue.cc:237] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04f25be9b1a4e81b053316a02d00790" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34569 } }
I20260620 02:03:16.493432 2953 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e04f25be9b1a4e81b053316a02d00790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04f25be9b1a4e81b053316a02d00790" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34569 } } }
I20260620 02:03:16.494040 2953 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:16.495739 2952 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e04f25be9b1a4e81b053316a02d00790. Latest consensus state: current_term: 1 leader_uuid: "e04f25be9b1a4e81b053316a02d00790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04f25be9b1a4e81b053316a02d00790" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 34569 } } }
I20260620 02:03:16.496157 2960 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:03:16.496241 2952 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790 [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:16.513540 2960 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:03:16.520928 2968 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:03:16.545925 2960 catalog_manager.cc:1382] Generated new cluster ID: 71456a61e18b44fd89100ba503a32a82
I20260620 02:03:16.546146 2960 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:03:16.580240 2960 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:03:16.582803 2960 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:03:16.603749 2960 catalog_manager.cc:6069] T 00000000000000000000000000000000 P e04f25be9b1a4e81b053316a02d00790: Generated new TSK 0
I20260620 02:03:16.605337 2960 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:16.745443 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.124s user 0.000s sys 0.026s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:16.876021 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.130s user 0.000s sys 0.025s
I20260620 02:03:16.876940 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:34569
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:17.596511 2974 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:17.596921 2974 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:17.597033 2974 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:17.617275 2974 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:17.617520 2974 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:03:17.639830 2974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:34569
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:17.645603 2974 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:17.653122 2974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:17.676628 2979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:17.676613 2980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:17.677820 2982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:17.679772 2974 server_base.cc:1061] running on GCE node
I20260620 02:03:17.681147 2974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:17.688484 2974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:17.696558 2974 hybrid_clock.cc:648] HybridClock initialized: now 1781920997696140 us; error 374 us; skew 500 ppm
I20260620 02:03:17.697252 2974 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:17 dist-test-slave-wg14 krb5kdc[2862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:17.704058 2974 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:03:17.706825 2974 webserver.cc:533] Webserver started at http://127.20.1.129:39955/ using document root <none> and password file <none>
I20260620 02:03:17.707753 2974 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:17.707882 2974 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:17.708225 2974 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:17.711748 2974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "e4489da50110498cb55f4254d61fdccb"
format_stamp: "Formatted at 2026-06-20 02:03:17 on dist-test-slave-wg14"
I20260620 02:03:17.713168 2974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "e4489da50110498cb55f4254d61fdccb"
format_stamp: "Formatted at 2026-06-20 02:03:17 on dist-test-slave-wg14"
I20260620 02:03:17.724776 2974 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.011s sys 0.000s
I20260620 02:03:17.737545 2989 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:17.740047 2974 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:03:17.740341 2974 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "e4489da50110498cb55f4254d61fdccb"
format_stamp: "Formatted at 2026-06-20 02:03:17 on dist-test-slave-wg14"
I20260620 02:03:17.740609 2974 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:17.793998 2974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:17.801345 2974 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:17.801690 2974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:17.808326 2974 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:03:17.810711 2974 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:03:17.810978 2974 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:17.811185 2974 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:03:17.811260 2974 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 02:03:17.894435 2974 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:37275
I20260620 02:03:17.894549 3102 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:37275 every 8 connection(s)
I20260620 02:03:17.896485 2974 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:03:17.905217 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 2974
I20260620 02:03:17.905452 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 02:03:17 dist-test-slave-wg14 krb5kdc[2862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:17.939368 3103 heartbeater.cc:344] Connected to a master server at 127.20.1.190:34569
I20260620 02:03:17.940052 3103 heartbeater.cc:461] Registering TS with master...
I20260620 02:03:17.941478 3103 heartbeater.cc:507] Master 127.20.1.190:34569 requested a full tablet report, sending...
I20260620 02:03:17.952117 2911 ts_manager.cc:194] Registered new tserver with Master: e4489da50110498cb55f4254d61fdccb (127.20.1.129:37275)
I20260620 02:03:17.954563 2911 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:51617
I20260620 02:03:17.967638 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Jun 20 02:03:18 dist-test-slave-wg14 krb5kdc[2862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:03:18 dist-test-slave-wg14 krb5kdc[2862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.20.1.190@KRBTEST.COM
I20260620 02:03:18.068589 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 2974
Jun 20 02:03:18 dist-test-slave-wg14 krb5kdc[2862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781920998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:03:18.101663 3098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:18.234288 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 2974
I20260620 02:03:18.255954 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 2878
I20260620 02:03:18.257948 2941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:18.414549 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 2878
2026-06-20T02:03:18Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/3 (6341 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/4
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:03:18 dist-test-slave-wg14 krb5kdc[3116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:03:18 dist-test-slave-wg14 krb5kdc[3116](info): set up 2 sockets
Jun 20 02:03:18 dist-test-slave-wg14 krb5kdc[3116](info): commencing operation
krb5kdc: starting...
W20260620 02:03:21.100800 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.618s user 0.004s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 02:03:21 dist-test-slave-wg14 krb5kdc[3116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:03:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:03:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:21.652503 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.190: real 0.102s user 0.003s sys 0.018s
I20260620 02:03:21.653803 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44599
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44599
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:22.336009 3132 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:22.336438 3132 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:22.336608 3132 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:22.347497 3132 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:03:22.347643 3132 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:22.347715 3132 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:03:22.347764 3132 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:03:22.365684 3132 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44599
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:22.367640 3132 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:22.369486 3132 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:22.388885 3138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:22.390934 3139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:22.392525 3141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:22.394876 3132 server_base.cc:1061] running on GCE node
I20260620 02:03:22.395872 3132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:22.397732 3132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:22.399364 3132 hybrid_clock.cc:648] HybridClock initialized: now 1781921002399248 us; error 64 us; skew 500 ppm
I20260620 02:03:22.400127 3132 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:22 dist-test-slave-wg14 krb5kdc[3116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921002, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:22.406841 3132 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:03:22.414112 3132 webserver.cc:533] Webserver started at http://127.20.1.190:42855/ using document root <none> and password file <none>
I20260620 02:03:22.415069 3132 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:22.415268 3132 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:22.415659 3132 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:22.419436 3132 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "89e7a3a5c9fc4323bd8f46542c7f2a88"
format_stamp: "Formatted at 2026-06-20 02:03:22 on dist-test-slave-wg14"
I20260620 02:03:22.420734 3132 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "89e7a3a5c9fc4323bd8f46542c7f2a88"
format_stamp: "Formatted at 2026-06-20 02:03:22 on dist-test-slave-wg14"
I20260620 02:03:22.435907 3132 fs_manager.cc:696] Time spent creating directory manager: real 0.015s user 0.011s sys 0.005s
I20260620 02:03:22.441452 3148 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:22.445137 3132 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:03:22.445402 3132 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "89e7a3a5c9fc4323bd8f46542c7f2a88"
format_stamp: "Formatted at 2026-06-20 02:03:22 on dist-test-slave-wg14"
I20260620 02:03:22.445672 3132 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:22.479547 3132 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:22.494733 3132 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:22.495043 3132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:22.545154 3199 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44599 every 8 connection(s)
I20260620 02:03:22.546041 3132 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44599
I20260620 02:03:22.553828 3132 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:03:22.554622 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 3132
I20260620 02:03:22.554867 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:03:22.561172 3200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:03:22 dist-test-slave-wg14 krb5kdc[3116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:22.586998 3200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88: Bootstrap starting.
I20260620 02:03:22.595834 3200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88: Neither blocks nor log segments found. Creating new log.
I20260620 02:03:22.598389 3200 log.cc:826] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88: Log is configured to *not* fsync() on all Append() calls
I20260620 02:03:22.604212 3200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88: No bootstrap required, opened a new log
I20260620 02:03:22.619719 3200 raft_consensus.cc:359] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e7a3a5c9fc4323bd8f46542c7f2a88" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44599 } }
I20260620 02:03:22.620491 3200 raft_consensus.cc:385] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:03:22.620615 3200 raft_consensus.cc:740] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e7a3a5c9fc4323bd8f46542c7f2a88, State: Initialized, Role: FOLLOWER
I20260620 02:03:22.621685 3200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e7a3a5c9fc4323bd8f46542c7f2a88" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44599 } }
I20260620 02:03:22.621990 3200 raft_consensus.cc:399] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:03:22.623380 3200 raft_consensus.cc:493] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:03:22.624568 3200 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:03:22.628450 3200 raft_consensus.cc:515] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e7a3a5c9fc4323bd8f46542c7f2a88" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44599 } }
I20260620 02:03:22.629479 3200 leader_election.cc:304] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89e7a3a5c9fc4323bd8f46542c7f2a88; no voters:
I20260620 02:03:22.631311 3200 leader_election.cc:290] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:03:22.631453 3205 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:03:22.635566 3205 raft_consensus.cc:697] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [term 1 LEADER]: Becoming Leader. State: Replica: 89e7a3a5c9fc4323bd8f46542c7f2a88, State: Running, Role: LEADER
I20260620 02:03:22.636759 3205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e7a3a5c9fc4323bd8f46542c7f2a88" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44599 } }
I20260620 02:03:22.637581 3200 sys_catalog.cc:565] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:03:22.640172 3206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89e7a3a5c9fc4323bd8f46542c7f2a88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e7a3a5c9fc4323bd8f46542c7f2a88" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44599 } } }
I20260620 02:03:22.640127 3207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89e7a3a5c9fc4323bd8f46542c7f2a88. Latest consensus state: current_term: 1 leader_uuid: "89e7a3a5c9fc4323bd8f46542c7f2a88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e7a3a5c9fc4323bd8f46542c7f2a88" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44599 } } }
I20260620 02:03:22.640636 3206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:22.640785 3207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88 [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:22.642506 3210 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:03:22.657027 3210 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:03:22.682209 3210 catalog_manager.cc:1382] Generated new cluster ID: 22ffdcd5d5a448d49be62c4a1be83b6a
I20260620 02:03:22.682372 3210 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:03:22.719028 3210 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:03:22.721640 3210 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:03:22.740880 3210 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 89e7a3a5c9fc4323bd8f46542c7f2a88: Generated new TSK 0
I20260620 02:03:22.743661 3210 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:22.879843 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.126s user 0.008s sys 0.023s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:23.019968 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.140s user 0.000s sys 0.038s
I20260620 02:03:23.021751 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44599
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:23.714313 3228 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:23.714725 3228 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:23.714850 3228 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:23.726322 3228 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:23.726541 3228 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:03:23.743932 3228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:44599
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:23.746220 3228 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:23.751376 3228 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:23.781250 3236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:23.781312 3233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:23.781291 3234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:23.783421 3228 server_base.cc:1061] running on GCE node
I20260620 02:03:23.788601 3228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:23.789975 3228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:23.791522 3228 hybrid_clock.cc:648] HybridClock initialized: now 1781921003791313 us; error 81 us; skew 500 ppm
I20260620 02:03:23.791993 3228 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:23 dist-test-slave-wg14 krb5kdc[3116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921003, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:23.799614 3228 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:03:23.802809 3228 webserver.cc:533] Webserver started at http://127.20.1.129:35637/ using document root <none> and password file <none>
I20260620 02:03:23.803922 3228 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:23.804098 3228 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:23.804549 3228 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:23.808112 3228 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "c82f122dc7624ef49a2776793890fabb"
format_stamp: "Formatted at 2026-06-20 02:03:23 on dist-test-slave-wg14"
I20260620 02:03:23.809163 3228 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "c82f122dc7624ef49a2776793890fabb"
format_stamp: "Formatted at 2026-06-20 02:03:23 on dist-test-slave-wg14"
I20260620 02:03:23.817369 3228 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260620 02:03:23.827382 3243 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:23.831394 3228 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.006s sys 0.000s
I20260620 02:03:23.831691 3228 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "c82f122dc7624ef49a2776793890fabb"
format_stamp: "Formatted at 2026-06-20 02:03:23 on dist-test-slave-wg14"
I20260620 02:03:23.831995 3228 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:23.861081 3228 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:23.867004 3228 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:23.867414 3228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:23.868769 3228 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:03:23.871754 3228 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:03:23.871923 3228 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:23.872085 3228 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:03:23.872157 3228 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:23.975675 3228 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35497
I20260620 02:03:23.976109 3356 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35497 every 8 connection(s)
I20260620 02:03:23.978691 3228 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:03:23.993196 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 3228
I20260620 02:03:23.993456 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 02:03:24 dist-test-slave-wg14 krb5kdc[3116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921003, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:24.030280 3357 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44599
I20260620 02:03:24.030885 3357 heartbeater.cc:461] Registering TS with master...
I20260620 02:03:24.033859 3357 heartbeater.cc:507] Master 127.20.1.190:44599 requested a full tablet report, sending...
I20260620 02:03:24.040740 3165 ts_manager.cc:194] Registered new tserver with Master: c82f122dc7624ef49a2776793890fabb (127.20.1.129:35497)
I20260620 02:03:24.043623 3165 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:52345
I20260620 02:03:24.052855 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Jun 20 02:03:24 dist-test-slave-wg14 krb5kdc[3116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:03:24 dist-test-slave-wg14 krb5kdc[3116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.20.1.190@KRBTEST.COM
I20260620 02:03:24.159550 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3228
Jun 20 02:03:24 dist-test-slave-wg14 krb5kdc[3116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:03:24.197192 3352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:24.371663 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 3228
I20260620 02:03:24.397068 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3132
I20260620 02:03:24.399066 3195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:24.605662 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 3132
2026-06-20T02:03:24Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/4 (6186 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/5
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:03:24 dist-test-slave-wg14 krb5kdc[3370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:03:24 dist-test-slave-wg14 krb5kdc[3370](info): set up 2 sockets
Jun 20 02:03:24 dist-test-slave-wg14 krb5kdc[3370](info): commencing operation
krb5kdc: starting...
W20260620 02:03:27.352650 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.684s user 0.007s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 02:03:27 dist-test-slave-wg14 krb5kdc[3370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:03:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:03:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:27.813187 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.190: real 0.113s user 0.000s sys 0.028s
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:27.917266 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.190: real 0.104s user 0.000s sys 0.022s
I20260620 02:03:27.918548 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44971
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:44971
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:28.617309 3386 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:28.618520 3386 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:28.618688 3386 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:28.630568 3386 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:03:28.630726 3386 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:28.630789 3386 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:03:28.630838 3386 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:03:28.649199 3386 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:44971
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:44971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:28.651335 3386 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:28.653188 3386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:28.666806 3392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:28.666837 3391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:28.671401 3394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:28.672847 3386 server_base.cc:1061] running on GCE node
I20260620 02:03:28.674634 3386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:28.678427 3386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:28.680449 3386 hybrid_clock.cc:648] HybridClock initialized: now 1781921008680246 us; error 139 us; skew 500 ppm
I20260620 02:03:28.681615 3386 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:28 dist-test-slave-wg14 krb5kdc[3370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921008, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:28.687403 3386 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:03:28.690059 3386 webserver.cc:533] Webserver started at http://127.20.1.190:37849/ using document root <none> and password file <none>
I20260620 02:03:28.691077 3386 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:28.691258 3386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:28.691668 3386 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:28.695600 3386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "229b54f3a49e4fda9a026dcabbe0732c"
format_stamp: "Formatted at 2026-06-20 02:03:28 on dist-test-slave-wg14"
I20260620 02:03:28.696785 3386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "229b54f3a49e4fda9a026dcabbe0732c"
format_stamp: "Formatted at 2026-06-20 02:03:28 on dist-test-slave-wg14"
I20260620 02:03:28.706025 3386 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.001s sys 0.008s
I20260620 02:03:28.712669 3401 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:28.716050 3386 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260620 02:03:28.716372 3386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "229b54f3a49e4fda9a026dcabbe0732c"
format_stamp: "Formatted at 2026-06-20 02:03:28 on dist-test-slave-wg14"
I20260620 02:03:28.716679 3386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:28.750795 3386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:28.761726 3386 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:28.762115 3386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:28.795091 3386 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:44971
I20260620 02:03:28.795168 3453 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:44971 every 8 connection(s)
I20260620 02:03:28.798103 3386 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:03:28.805311 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 3386
I20260620 02:03:28.805497 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:03:28.805900 3454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:03:28.837250 3454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c: Bootstrap starting.
Jun 20 02:03:28 dist-test-slave-wg14 krb5kdc[3370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:28.842384 3454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c: Neither blocks nor log segments found. Creating new log.
I20260620 02:03:28.844830 3454 log.cc:826] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:03:28.855635 3454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c: No bootstrap required, opened a new log
I20260620 02:03:28.875419 3454 raft_consensus.cc:359] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229b54f3a49e4fda9a026dcabbe0732c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44971 } }
I20260620 02:03:28.875922 3454 raft_consensus.cc:385] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:03:28.876019 3454 raft_consensus.cc:740] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229b54f3a49e4fda9a026dcabbe0732c, State: Initialized, Role: FOLLOWER
I20260620 02:03:28.877223 3454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229b54f3a49e4fda9a026dcabbe0732c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44971 } }
I20260620 02:03:28.877565 3454 raft_consensus.cc:399] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:03:28.877784 3454 raft_consensus.cc:493] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:03:28.878083 3454 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:03:28.881078 3454 raft_consensus.cc:515] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229b54f3a49e4fda9a026dcabbe0732c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44971 } }
I20260620 02:03:28.881999 3454 leader_election.cc:304] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 229b54f3a49e4fda9a026dcabbe0732c; no voters:
I20260620 02:03:28.882884 3454 leader_election.cc:290] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:03:28.883283 3459 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:03:28.887748 3459 raft_consensus.cc:697] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [term 1 LEADER]: Becoming Leader. State: Replica: 229b54f3a49e4fda9a026dcabbe0732c, State: Running, Role: LEADER
I20260620 02:03:28.893683 3459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229b54f3a49e4fda9a026dcabbe0732c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44971 } }
I20260620 02:03:28.896006 3454 sys_catalog.cc:565] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:03:28.898988 3460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 229b54f3a49e4fda9a026dcabbe0732c. Latest consensus state: current_term: 1 leader_uuid: "229b54f3a49e4fda9a026dcabbe0732c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229b54f3a49e4fda9a026dcabbe0732c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44971 } } }
I20260620 02:03:28.901017 3460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:28.899070 3461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "229b54f3a49e4fda9a026dcabbe0732c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229b54f3a49e4fda9a026dcabbe0732c" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 44971 } } }
I20260620 02:03:28.901895 3461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:28.912184 3464 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:03:28.924623 3464 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:03:28.944964 3464 catalog_manager.cc:1382] Generated new cluster ID: 12c50e3d6dc44f13b3e6afdbbbc3e646
I20260620 02:03:28.945160 3464 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:03:28.962703 3464 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:03:28.973634 3464 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:03:29.012686 3464 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 229b54f3a49e4fda9a026dcabbe0732c: Generated new TSK 0
I20260620 02:03:29.014058 3464 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:29.139550 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.103s user 0.000s sys 0.022s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:29.242274 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.103s user 0.000s sys 0.021s
I20260620 02:03:29.243201 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:44971
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:29.819871 3482 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:29.820274 3482 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:29.820386 3482 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:29.830827 3482 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:29.831081 3482 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:03:29.847589 3482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:44971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:29.849597 3482 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:29.851430 3482 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:29.869529 3488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:29.869536 3487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:29.871588 3482 server_base.cc:1061] running on GCE node
W20260620 02:03:29.873330 3490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:29.875484 3482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:29.880321 3482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:29.883948 3482 hybrid_clock.cc:648] HybridClock initialized: now 1781921009883760 us; error 133 us; skew 500 ppm
I20260620 02:03:29.884496 3482 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:29 dist-test-slave-wg14 krb5kdc[3370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921009, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:29.888727 3482 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:03:29.891301 3482 webserver.cc:533] Webserver started at http://127.20.1.129:36005/ using document root <none> and password file <none>
I20260620 02:03:29.892351 3482 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:29.892489 3482 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:29.893179 3482 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:29.898168 3482 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "33e27eb92699490f95359cec66aab908"
format_stamp: "Formatted at 2026-06-20 02:03:29 on dist-test-slave-wg14"
I20260620 02:03:29.899662 3482 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "33e27eb92699490f95359cec66aab908"
format_stamp: "Formatted at 2026-06-20 02:03:29 on dist-test-slave-wg14"
I20260620 02:03:29.908326 3482 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260620 02:03:29.914376 3497 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:29.919713 3482 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.009s sys 0.000s
I20260620 02:03:29.920742 3482 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "33e27eb92699490f95359cec66aab908"
format_stamp: "Formatted at 2026-06-20 02:03:29 on dist-test-slave-wg14"
I20260620 02:03:29.921103 3482 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:29.940865 3482 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:29.946956 3482 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:29.947474 3482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:29.948824 3482 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:03:29.951164 3482 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:03:29.951297 3482 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 02:03:29.951447 3482 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:03:29.951505 3482 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:30.011226 3482 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:44449
I20260620 02:03:30.011327 3610 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:44449 every 8 connection(s)
I20260620 02:03:30.013026 3482 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:03:30.022244 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 3482
I20260620 02:03:30.022475 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 02:03:30 dist-test-slave-wg14 krb5kdc[3370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921009, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:30.053218 3611 heartbeater.cc:344] Connected to a master server at 127.20.1.190:44971
I20260620 02:03:30.054323 3611 heartbeater.cc:461] Registering TS with master...
I20260620 02:03:30.058506 3611 heartbeater.cc:507] Master 127.20.1.190:44971 requested a full tablet report, sending...
I20260620 02:03:30.063578 3418 ts_manager.cc:194] Registered new tserver with Master: 33e27eb92699490f95359cec66aab908 (127.20.1.129:44449)
I20260620 02:03:30.066093 3418 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:50983
I20260620 02:03:30.070534 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Jun 20 02:03:30 dist-test-slave-wg14 krb5kdc[3370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:03:30 dist-test-slave-wg14 krb5kdc[3370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.20.1.190@KRBTEST.COM
I20260620 02:03:30.184432 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3482
Jun 20 02:03:30 dist-test-slave-wg14 krb5kdc[3370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:03:30.211334 3606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:30.361645 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 3482
I20260620 02:03:30.389569 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3386
I20260620 02:03:30.392051 3449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:30.563570 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 3386
2026-06-20T02:03:30Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/5 (5946 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/6
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:03:30 dist-test-slave-wg14 krb5kdc[3624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:03:30 dist-test-slave-wg14 krb5kdc[3624](info): set up 2 sockets
Jun 20 02:03:30 dist-test-slave-wg14 krb5kdc[3624](info): commencing operation
krb5kdc: starting...
W20260620 02:03:33.532102 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.918s user 0.005s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 02:03:33 dist-test-slave-wg14 krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:03:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:03:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:34.008858 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.190: real 0.105s user 0.001s sys 0.019s
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:03:34.107227 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:46563
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:34445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:46563
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:34.826710 3641 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:34.828131 3641 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:34.828286 3641 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:34.845773 3641 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:03:34.845957 3641 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:34.846022 3641 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:03:34.846063 3641 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:03:34.864858 3641 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:34445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:46563
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:46563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:34.867425 3641 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:34.869782 3641 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:34.894079 3647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:34.894620 3646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:34.895051 3649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:34.895327 3641 server_base.cc:1061] running on GCE node
I20260620 02:03:34.897156 3641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:34.899235 3641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:34.902918 3641 hybrid_clock.cc:648] HybridClock initialized: now 1781921014902487 us; error 357 us; skew 500 ppm
I20260620 02:03:34.906332 3641 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:34 dist-test-slave-wg14 krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921014, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:34.912674 3641 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:03:34.916239 3641 webserver.cc:533] Webserver started at http://127.20.1.190:35329/ using document root <none> and password file <none>
I20260620 02:03:34.917333 3641 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:34.917497 3641 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:34.917870 3641 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:34.921788 3641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "7cfd47c261624e5ab80b8db21b7f68db"
format_stamp: "Formatted at 2026-06-20 02:03:34 on dist-test-slave-wg14"
I20260620 02:03:34.922845 3641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "7cfd47c261624e5ab80b8db21b7f68db"
format_stamp: "Formatted at 2026-06-20 02:03:34 on dist-test-slave-wg14"
I20260620 02:03:34.935351 3641 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.008s sys 0.000s
I20260620 02:03:34.942123 3656 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:34.944356 3641 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 02:03:34.944602 3641 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "7cfd47c261624e5ab80b8db21b7f68db"
format_stamp: "Formatted at 2026-06-20 02:03:34 on dist-test-slave-wg14"
I20260620 02:03:34.946996 3641 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:35.005249 3641 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:35.030305 3641 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:35.030611 3641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:35.079993 3708 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:46563 every 8 connection(s)
I20260620 02:03:35.079816 3641 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:46563
I20260620 02:03:35.083036 3641 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:03:35.089210 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 3641
I20260620 02:03:35.089484 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:03:35.096985 3709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:03:35 dist-test-slave-wg14 krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:35.115513 3709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db: Bootstrap starting.
I20260620 02:03:35.124528 3709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db: Neither blocks nor log segments found. Creating new log.
I20260620 02:03:35.126699 3709 log.cc:826] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db: Log is configured to *not* fsync() on all Append() calls
I20260620 02:03:35.138223 3709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db: No bootstrap required, opened a new log
I20260620 02:03:35.147626 3709 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfd47c261624e5ab80b8db21b7f68db" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46563 } }
I20260620 02:03:35.148121 3709 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:03:35.148249 3709 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cfd47c261624e5ab80b8db21b7f68db, State: Initialized, Role: FOLLOWER
I20260620 02:03:35.149119 3709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfd47c261624e5ab80b8db21b7f68db" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46563 } }
I20260620 02:03:35.149420 3709 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:03:35.149582 3709 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:03:35.149806 3709 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:03:35.153132 3709 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfd47c261624e5ab80b8db21b7f68db" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46563 } }
I20260620 02:03:35.153920 3709 leader_election.cc:304] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cfd47c261624e5ab80b8db21b7f68db; no voters:
I20260620 02:03:35.156247 3709 leader_election.cc:290] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:03:35.156538 3714 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:03:35.161289 3709 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:03:35.164326 3714 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [term 1 LEADER]: Becoming Leader. State: Replica: 7cfd47c261624e5ab80b8db21b7f68db, State: Running, Role: LEADER
I20260620 02:03:35.165547 3714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfd47c261624e5ab80b8db21b7f68db" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46563 } }
I20260620 02:03:35.174396 3716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cfd47c261624e5ab80b8db21b7f68db. Latest consensus state: current_term: 1 leader_uuid: "7cfd47c261624e5ab80b8db21b7f68db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfd47c261624e5ab80b8db21b7f68db" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46563 } } }
I20260620 02:03:35.175429 3716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:35.174933 3715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cfd47c261624e5ab80b8db21b7f68db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfd47c261624e5ab80b8db21b7f68db" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46563 } } }
I20260620 02:03:35.182636 3715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:35.203557 3724 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:03:35.227679 3724 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:03:35.254149 3724 catalog_manager.cc:1382] Generated new cluster ID: 2347baab5f1f44569516592cd1fa1845
I20260620 02:03:35.254343 3724 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:03:35.287051 3724 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:03:35.294627 3724 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:03:35.331336 3724 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 7cfd47c261624e5ab80b8db21b7f68db: Generated new TSK 0
I20260620 02:03:35.332998 3724 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:35.526216 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.165s user 0.000s sys 0.026s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:35.669456 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.143s user 0.006s sys 0.026s
I20260620 02:03:35.670753 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:46563
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:34445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:36.514257 3737 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:36.514752 3737 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:36.514946 3737 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:36.526576 3737 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:36.526878 3737 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:03:36.546756 3737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:34445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:46563
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:36.549739 3737 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:36.553210 3737 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:36.574951 3742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:36.576592 3743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:36.577476 3745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:36.580565 3737 server_base.cc:1061] running on GCE node
I20260620 02:03:36.581660 3737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:36.583160 3737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:36.589406 3737 hybrid_clock.cc:648] HybridClock initialized: now 1781921016589236 us; error 116 us; skew 500 ppm
I20260620 02:03:36.589936 3737 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:36 dist-test-slave-wg14 krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921016, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:36.595468 3737 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:03:36.597880 3737 webserver.cc:533] Webserver started at http://127.20.1.129:34701/ using document root <none> and password file <none>
I20260620 02:03:36.600270 3737 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:36.600477 3737 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:36.601707 3737 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:36.606876 3737 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "d3722176e2a94635975dc46fbd6b7561"
format_stamp: "Formatted at 2026-06-20 02:03:36 on dist-test-slave-wg14"
I20260620 02:03:36.608420 3737 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "d3722176e2a94635975dc46fbd6b7561"
format_stamp: "Formatted at 2026-06-20 02:03:36 on dist-test-slave-wg14"
I20260620 02:03:36.616294 3737 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260620 02:03:36.628135 3752 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:36.635900 3737 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.003s sys 0.001s
I20260620 02:03:36.636164 3737 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "d3722176e2a94635975dc46fbd6b7561"
format_stamp: "Formatted at 2026-06-20 02:03:36 on dist-test-slave-wg14"
I20260620 02:03:36.636418 3737 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:36.703589 3737 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:36.712653 3737 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:36.713055 3737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:36.714388 3737 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:03:36.717980 3737 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:03:36.718132 3737 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:36.718317 3737 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:03:36.718375 3737 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:36.840788 3737 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:34647
I20260620 02:03:36.840828 3865 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:34647 every 8 connection(s)
I20260620 02:03:36.843026 3737 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:03:36.850440 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 3737
I20260620 02:03:36.850666 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 02:03:36 dist-test-slave-wg14 krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921016, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:36.899076 3866 heartbeater.cc:344] Connected to a master server at 127.20.1.190:46563
I20260620 02:03:36.899590 3866 heartbeater.cc:461] Registering TS with master...
I20260620 02:03:36.900589 3866 heartbeater.cc:507] Master 127.20.1.190:46563 requested a full tablet report, sending...
I20260620 02:03:36.904458 3673 ts_manager.cc:194] Registered new tserver with Master: d3722176e2a94635975dc46fbd6b7561 (127.20.1.129:34647)
I20260620 02:03:36.907491 3673 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:40271
I20260620 02:03:36.912778 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Jun 20 02:03:36 dist-test-slave-wg14 krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:03:37 dist-test-slave-wg14 krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.20.1.190@KRBTEST.COM
I20260620 02:03:37.016978 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3737
Jun 20 02:03:37 dist-test-slave-wg14 krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:03:37.054957 3861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:37.207083 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 3737
I20260620 02:03:37.237950 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3641
I20260620 02:03:37.239571 3704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:37.413421 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 3641
2026-06-20T02:03:37Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/6 (6856 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/7
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:03:37 dist-test-slave-wg14 krb5kdc[3879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:03:37 dist-test-slave-wg14 krb5kdc[3879](info): set up 2 sockets
Jun 20 02:03:37 dist-test-slave-wg14 krb5kdc[3879](info): commencing operation
krb5kdc: starting...
W20260620 02:03:40.473260 20486 mini_kdc.cc:121] Time spent starting KDC: real 3.002s user 0.010s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 02:03:40 dist-test-slave-wg14 krb5kdc[3879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:03:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:03:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:40.956517 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.190: real 0.115s user 0.004s sys 0.024s
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:41.092624 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.190: real 0.136s user 0.004s sys 0.024s
I20260620 02:03:41.094179 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:37369
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:44789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:37369
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:41.804040 3895 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:41.804690 3895 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:41.804826 3895 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:41.826221 3895 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:03:41.826391 3895 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:41.826450 3895 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:03:41.826493 3895 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:03:41.844835 3895 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:44789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:37369
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:37369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:41.846789 3895 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:41.848553 3895 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:41.873724 3901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:41.874836 3900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:41.876456 3903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:41.878531 3895 server_base.cc:1061] running on GCE node
I20260620 02:03:41.880976 3895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:41.885931 3895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:41.889549 3895 hybrid_clock.cc:648] HybridClock initialized: now 1781921021889427 us; error 79 us; skew 500 ppm
I20260620 02:03:41.890146 3895 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:41 dist-test-slave-wg14 krb5kdc[3879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921021, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:41.898468 3895 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:03:41.905102 3895 webserver.cc:533] Webserver started at http://127.20.1.190:35567/ using document root <none> and password file <none>
I20260620 02:03:41.906220 3895 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:41.906409 3895 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:41.906826 3895 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:41.910238 3895 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "c28397776fa4433782e63f9923ef92b3"
format_stamp: "Formatted at 2026-06-20 02:03:41 on dist-test-slave-wg14"
I20260620 02:03:41.911702 3895 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "c28397776fa4433782e63f9923ef92b3"
format_stamp: "Formatted at 2026-06-20 02:03:41 on dist-test-slave-wg14"
I20260620 02:03:41.932963 3895 fs_manager.cc:696] Time spent creating directory manager: real 0.021s user 0.018s sys 0.001s
I20260620 02:03:41.942517 3910 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:41.946503 3895 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260620 02:03:41.946875 3895 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "c28397776fa4433782e63f9923ef92b3"
format_stamp: "Formatted at 2026-06-20 02:03:41 on dist-test-slave-wg14"
I20260620 02:03:41.947403 3895 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:41.978130 3895 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:41.996687 3895 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:41.997503 3895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:42.044987 3895 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:37369
I20260620 02:03:42.045399 3962 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:37369 every 8 connection(s)
I20260620 02:03:42.047324 3895 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:03:42.050041 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 3895
I20260620 02:03:42.050240 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:03:42.056546 3963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:03:42 dist-test-slave-wg14 krb5kdc[3879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:42.080137 3963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3: Bootstrap starting.
I20260620 02:03:42.087867 3963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3: Neither blocks nor log segments found. Creating new log.
I20260620 02:03:42.094348 3963 log.cc:826] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3: Log is configured to *not* fsync() on all Append() calls
I20260620 02:03:42.100175 3963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3: No bootstrap required, opened a new log
I20260620 02:03:42.116617 3963 raft_consensus.cc:359] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28397776fa4433782e63f9923ef92b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 37369 } }
I20260620 02:03:42.117177 3963 raft_consensus.cc:385] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:03:42.117316 3963 raft_consensus.cc:740] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c28397776fa4433782e63f9923ef92b3, State: Initialized, Role: FOLLOWER
I20260620 02:03:42.118113 3963 consensus_queue.cc:260] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28397776fa4433782e63f9923ef92b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 37369 } }
I20260620 02:03:42.118407 3963 raft_consensus.cc:399] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:03:42.118561 3963 raft_consensus.cc:493] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:03:42.118738 3963 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:03:42.121438 3963 raft_consensus.cc:515] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28397776fa4433782e63f9923ef92b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 37369 } }
I20260620 02:03:42.122207 3963 leader_election.cc:304] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c28397776fa4433782e63f9923ef92b3; no voters:
I20260620 02:03:42.123073 3963 leader_election.cc:290] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:03:42.123531 3968 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:03:42.124436 3968 raft_consensus.cc:697] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [term 1 LEADER]: Becoming Leader. State: Replica: c28397776fa4433782e63f9923ef92b3, State: Running, Role: LEADER
I20260620 02:03:42.125551 3968 consensus_queue.cc:237] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28397776fa4433782e63f9923ef92b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 37369 } }
I20260620 02:03:42.131270 3963 sys_catalog.cc:565] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:03:42.134193 3970 sys_catalog.cc:455] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c28397776fa4433782e63f9923ef92b3. Latest consensus state: current_term: 1 leader_uuid: "c28397776fa4433782e63f9923ef92b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28397776fa4433782e63f9923ef92b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 37369 } } }
I20260620 02:03:42.135229 3970 sys_catalog.cc:458] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:42.134687 3969 sys_catalog.cc:455] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c28397776fa4433782e63f9923ef92b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28397776fa4433782e63f9923ef92b3" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 37369 } } }
I20260620 02:03:42.136185 3969 sys_catalog.cc:458] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:42.137053 3973 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:03:42.148772 3973 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:03:42.176774 3973 catalog_manager.cc:1382] Generated new cluster ID: 85b58e502587451b932ef17a9503924a
I20260620 02:03:42.176977 3973 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:03:42.230492 3973 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:03:42.233255 3973 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:03:42.257342 3973 catalog_manager.cc:6069] T 00000000000000000000000000000000 P c28397776fa4433782e63f9923ef92b3: Generated new TSK 0
I20260620 02:03:42.258652 3973 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:42.399111 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.126s user 0.001s sys 0.038s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:42.518705 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.119s user 0.007s sys 0.022s
I20260620 02:03:42.519718 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:37369
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:44789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:43.151748 3991 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:43.152468 3991 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:43.152995 3991 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:43.167815 3991 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:43.168138 3991 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:03:43.188527 3991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:44789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:37369
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:43.190358 3991 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:43.193105 3991 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:43.222180 3996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:43.221866 3997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:43.223392 3999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:43.228292 3991 server_base.cc:1061] running on GCE node
I20260620 02:03:43.229283 3991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:43.231662 3991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:43.235373 3991 hybrid_clock.cc:648] HybridClock initialized: now 1781921023235169 us; error 155 us; skew 500 ppm
I20260620 02:03:43.236215 3991 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:43 dist-test-slave-wg14 krb5kdc[3879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:43.240717 3991 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:03:43.243566 3991 webserver.cc:533] Webserver started at http://127.20.1.129:41397/ using document root <none> and password file <none>
I20260620 02:03:43.244908 3991 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:43.245141 3991 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:43.245633 3991 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:43.248906 3991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "b8d7e99aeb704977923d656b44df166a"
format_stamp: "Formatted at 2026-06-20 02:03:43 on dist-test-slave-wg14"
I20260620 02:03:43.249933 3991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "b8d7e99aeb704977923d656b44df166a"
format_stamp: "Formatted at 2026-06-20 02:03:43 on dist-test-slave-wg14"
I20260620 02:03:43.260582 3991 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s
I20260620 02:03:43.268540 4006 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:43.270637 3991 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:03:43.270934 3991 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "b8d7e99aeb704977923d656b44df166a"
format_stamp: "Formatted at 2026-06-20 02:03:43 on dist-test-slave-wg14"
I20260620 02:03:43.271257 3991 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:43.323385 3991 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:43.331054 3991 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:43.331651 3991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:43.333664 3991 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:03:43.338598 3991 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:03:43.338722 3991 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:43.338832 3991 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:03:43.338900 3991 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:43.399948 3991 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:46869
I20260620 02:03:43.399994 4119 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:46869 every 8 connection(s)
I20260620 02:03:43.402022 3991 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:03:43.403447 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 3991
I20260620 02:03:43.403685 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 02:03:43 dist-test-slave-wg14 krb5kdc[3879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:43.442924 4120 heartbeater.cc:344] Connected to a master server at 127.20.1.190:37369
I20260620 02:03:43.444326 4120 heartbeater.cc:461] Registering TS with master...
I20260620 02:03:43.445714 4120 heartbeater.cc:507] Master 127.20.1.190:37369 requested a full tablet report, sending...
I20260620 02:03:43.448874 3928 ts_manager.cc:194] Registered new tserver with Master: b8d7e99aeb704977923d656b44df166a (127.20.1.129:46869)
I20260620 02:03:43.451056 3928 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:39867
I20260620 02:03:43.456938 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Jun 20 02:03:43 dist-test-slave-wg14 krb5kdc[3879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:03:43 dist-test-slave-wg14 krb5kdc[3879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.20.1.190@KRBTEST.COM
I20260620 02:03:43.537703 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3991
Jun 20 02:03:43 dist-test-slave-wg14 krb5kdc[3879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:03:43.569341 4115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:43.694222 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 3991
I20260620 02:03:43.716779 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 3895
I20260620 02:03:43.718967 3958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:43.890182 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 3895
2026-06-20T02:03:43Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/7 (6474 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/8
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:03:44 dist-test-slave-wg14 krb5kdc[4133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:03:44 dist-test-slave-wg14 krb5kdc[4133](info): set up 2 sockets
Jun 20 02:03:44 dist-test-slave-wg14 krb5kdc[4133](info): commencing operation
krb5kdc: starting...
W20260620 02:03:47.237285 20486 mini_kdc.cc:121] Time spent starting KDC: real 3.290s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 02:03:47 dist-test-slave-wg14 krb5kdc[4133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:03:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:03:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:47.722609 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.190: real 0.109s user 0.001s sys 0.023s
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:47.835346 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.190: real 0.113s user 0.004s sys 0.019s
I20260620 02:03:47.836481 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:37769
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:40559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:37769
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:48.632680 4149 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:48.633078 4149 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:48.633172 4149 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:48.645032 4149 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:03:48.645201 4149 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:48.645272 4149 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:03:48.645329 4149 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:03:48.675823 4149 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:37769
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:37769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:48.678238 4149 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:48.680296 4149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:48.702791 4154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:48.702791 4155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:48.704911 4157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:48.707834 4149 server_base.cc:1061] running on GCE node
I20260620 02:03:48.711789 4149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:48.716058 4149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:48.717468 4149 hybrid_clock.cc:648] HybridClock initialized: now 1781921028717416 us; error 69 us; skew 500 ppm
I20260620 02:03:48.718082 4149 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:48 dist-test-slave-wg14 krb5kdc[4133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921028, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:48.722898 4149 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:03:48.725623 4149 webserver.cc:533] Webserver started at http://127.20.1.190:34471/ using document root <none> and password file <none>
I20260620 02:03:48.726670 4149 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:48.726843 4149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:48.727887 4149 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:48.734408 4149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "4fe5b48fc3fd4e74bde3b72fd64a3adf"
format_stamp: "Formatted at 2026-06-20 02:03:48 on dist-test-slave-wg14"
I20260620 02:03:48.735908 4149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "4fe5b48fc3fd4e74bde3b72fd64a3adf"
format_stamp: "Formatted at 2026-06-20 02:03:48 on dist-test-slave-wg14"
I20260620 02:03:48.748203 4149 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.007s sys 0.004s
I20260620 02:03:48.756871 4164 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:48.759842 4149 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.004s sys 0.001s
I20260620 02:03:48.760421 4149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "4fe5b48fc3fd4e74bde3b72fd64a3adf"
format_stamp: "Formatted at 2026-06-20 02:03:48 on dist-test-slave-wg14"
I20260620 02:03:48.761155 4149 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:48.798405 4149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:48.804404 4149 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:48.804759 4149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:48.851598 4149 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:37769
I20260620 02:03:48.851804 4216 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:37769 every 8 connection(s)
I20260620 02:03:48.854151 4149 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:03:48.860348 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 4149
I20260620 02:03:48.860580 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:03:48.861316 4217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:03:48.896413 4217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf: Bootstrap starting.
Jun 20 02:03:48 dist-test-slave-wg14 krb5kdc[4133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:48.902352 4217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf: Neither blocks nor log segments found. Creating new log.
I20260620 02:03:48.904067 4217 log.cc:826] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf: Log is configured to *not* fsync() on all Append() calls
I20260620 02:03:48.914978 4217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf: No bootstrap required, opened a new log
I20260620 02:03:48.922394 4217 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe5b48fc3fd4e74bde3b72fd64a3adf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 37769 } }
I20260620 02:03:48.922978 4217 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:03:48.923190 4217 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe5b48fc3fd4e74bde3b72fd64a3adf, State: Initialized, Role: FOLLOWER
I20260620 02:03:48.924119 4217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe5b48fc3fd4e74bde3b72fd64a3adf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 37769 } }
I20260620 02:03:48.924392 4217 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:03:48.924551 4217 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:03:48.924706 4217 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:03:48.930608 4217 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe5b48fc3fd4e74bde3b72fd64a3adf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 37769 } }
I20260620 02:03:48.934077 4217 leader_election.cc:304] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fe5b48fc3fd4e74bde3b72fd64a3adf; no voters:
I20260620 02:03:48.936209 4217 leader_election.cc:290] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:03:48.936414 4222 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:03:48.937871 4222 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [term 1 LEADER]: Becoming Leader. State: Replica: 4fe5b48fc3fd4e74bde3b72fd64a3adf, State: Running, Role: LEADER
I20260620 02:03:48.938781 4222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe5b48fc3fd4e74bde3b72fd64a3adf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 37769 } }
I20260620 02:03:48.940409 4217 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:03:48.943411 4223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fe5b48fc3fd4e74bde3b72fd64a3adf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe5b48fc3fd4e74bde3b72fd64a3adf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 37769 } } }
I20260620 02:03:48.944492 4223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:48.944190 4224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fe5b48fc3fd4e74bde3b72fd64a3adf. Latest consensus state: current_term: 1 leader_uuid: "4fe5b48fc3fd4e74bde3b72fd64a3adf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe5b48fc3fd4e74bde3b72fd64a3adf" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 37769 } } }
I20260620 02:03:48.944662 4224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:48.951182 4228 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:03:48.963335 4228 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:03:48.996435 4228 catalog_manager.cc:1382] Generated new cluster ID: 9a614b3a933f4e528290d730ae65ec3f
I20260620 02:03:48.996834 4228 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:03:49.050693 4228 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:03:49.052786 4228 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:03:49.071969 4228 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 4fe5b48fc3fd4e74bde3b72fd64a3adf: Generated new TSK 0
I20260620 02:03:49.073230 4228 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:49.210942 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.115s user 0.000s sys 0.043s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:49.339430 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.128s user 0.000s sys 0.053s
I20260620 02:03:49.340533 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:37769
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:40559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:49.964741 4245 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:49.965384 4245 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:49.965960 4245 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:49.978127 4245 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:49.978425 4245 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:03:50.005323 4245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:37769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:50.007655 4245 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:50.016678 4245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:50.035392 4251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:50.035468 4250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:50.036273 4253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:50.039611 4245 server_base.cc:1061] running on GCE node
I20260620 02:03:50.040714 4245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:50.042048 4245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:50.043625 4245 hybrid_clock.cc:648] HybridClock initialized: now 1781921030043428 us; error 75 us; skew 500 ppm
I20260620 02:03:50.044811 4245 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:50 dist-test-slave-wg14 krb5kdc[4133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921030, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:50.049510 4245 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:03:50.052511 4245 webserver.cc:533] Webserver started at http://127.20.1.129:35797/ using document root <none> and password file <none>
I20260620 02:03:50.053493 4245 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:50.053644 4245 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:50.054003 4245 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:50.058350 4245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "3e44ff871dae45cbae5cf1f6d605cc57"
format_stamp: "Formatted at 2026-06-20 02:03:50 on dist-test-slave-wg14"
I20260620 02:03:50.059836 4245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "3e44ff871dae45cbae5cf1f6d605cc57"
format_stamp: "Formatted at 2026-06-20 02:03:50 on dist-test-slave-wg14"
I20260620 02:03:50.070350 4245 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.006s sys 0.003s
I20260620 02:03:50.084408 4260 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:50.086915 4245 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.008s sys 0.000s
I20260620 02:03:50.087234 4245 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "3e44ff871dae45cbae5cf1f6d605cc57"
format_stamp: "Formatted at 2026-06-20 02:03:50 on dist-test-slave-wg14"
I20260620 02:03:50.087489 4245 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:50.111449 4245 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:50.121307 4245 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:50.121718 4245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:50.123237 4245 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:03:50.125470 4245 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:03:50.125587 4245 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:50.125694 4245 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:03:50.125731 4245 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:50.189553 4245 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:42953
I20260620 02:03:50.189920 4373 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:42953 every 8 connection(s)
I20260620 02:03:50.191301 4245 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:03:50.201984 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 4245
I20260620 02:03:50.202710 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 02:03:50 dist-test-slave-wg14 krb5kdc[4133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921030, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:50.240001 4374 heartbeater.cc:344] Connected to a master server at 127.20.1.190:37769
I20260620 02:03:50.241415 4374 heartbeater.cc:461] Registering TS with master...
I20260620 02:03:50.244143 4374 heartbeater.cc:507] Master 127.20.1.190:37769 requested a full tablet report, sending...
I20260620 02:03:50.248400 4181 ts_manager.cc:194] Registered new tserver with Master: 3e44ff871dae45cbae5cf1f6d605cc57 (127.20.1.129:42953)
I20260620 02:03:50.250578 4181 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:54183
I20260620 02:03:50.258486 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Jun 20 02:03:50 dist-test-slave-wg14 krb5kdc[4133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:03:50 dist-test-slave-wg14 krb5kdc[4133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.20.1.190@KRBTEST.COM
I20260620 02:03:50.339998 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4245
Jun 20 02:03:50 dist-test-slave-wg14 krb5kdc[4133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:03:50.369001 4369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:50.499537 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 4245
I20260620 02:03:50.518548 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4149
I20260620 02:03:50.520465 4212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:50.669575 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 4149
2026-06-20T02:03:50Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/8 (6780 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/9
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:03:50 dist-test-slave-wg14 krb5kdc[4387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:03:50 dist-test-slave-wg14 krb5kdc[4387](info): set up 2 sockets
Jun 20 02:03:50 dist-test-slave-wg14 krb5kdc[4387](info): commencing operation
krb5kdc: starting...
W20260620 02:03:53.560434 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.834s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 02:03:53 dist-test-slave-wg14 krb5kdc[4387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:03:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:03:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:54.134613 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.190: real 0.115s user 0.000s sys 0.026s
I20260620 02:03:54.136082 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:33699
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:40771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:33699
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:54.909348 4403 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:54.909790 4403 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:54.909935 4403 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:54.922763 4403 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:03:54.922945 4403 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:54.923036 4403 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:03:54.923110 4403 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:03:54.953886 4403 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:33699
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:33699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:54.958132 4403 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:54.960088 4403 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:54.977139 4408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:54.979198 4409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:54.982609 4411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:54.983352 4403 server_base.cc:1061] running on GCE node
I20260620 02:03:54.984457 4403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:54.991037 4403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:54.995712 4403 hybrid_clock.cc:648] HybridClock initialized: now 1781921034995564 us; error 97 us; skew 500 ppm
I20260620 02:03:54.996340 4403 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:54 dist-test-slave-wg14 krb5kdc[4387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921034, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:55.004004 4403 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:03:55.008229 4403 webserver.cc:533] Webserver started at http://127.20.1.190:38513/ using document root <none> and password file <none>
I20260620 02:03:55.009565 4403 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:55.009764 4403 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:55.010344 4403 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:55.014415 4403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "5c38084da9df43a2bae44581a93a1f00"
format_stamp: "Formatted at 2026-06-20 02:03:55 on dist-test-slave-wg14"
I20260620 02:03:55.015763 4403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "5c38084da9df43a2bae44581a93a1f00"
format_stamp: "Formatted at 2026-06-20 02:03:55 on dist-test-slave-wg14"
I20260620 02:03:55.032037 4403 fs_manager.cc:696] Time spent creating directory manager: real 0.016s user 0.010s sys 0.005s
I20260620 02:03:55.040732 4418 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:55.044133 4403 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:03:55.044449 4403 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "5c38084da9df43a2bae44581a93a1f00"
format_stamp: "Formatted at 2026-06-20 02:03:55 on dist-test-slave-wg14"
I20260620 02:03:55.045336 4403 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:55.088896 4403 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:55.102402 4403 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:55.102742 4403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:55.147317 4403 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:33699
I20260620 02:03:55.147334 4470 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:33699 every 8 connection(s)
I20260620 02:03:55.149219 4403 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:03:55.154429 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 4403
I20260620 02:03:55.154835 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:03:55.159406 4471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:03:55.181512 4471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00: Bootstrap starting.
Jun 20 02:03:55 dist-test-slave-wg14 krb5kdc[4387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:55.188625 4471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00: Neither blocks nor log segments found. Creating new log.
I20260620 02:03:55.192481 4471 log.cc:826] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00: Log is configured to *not* fsync() on all Append() calls
I20260620 02:03:55.201786 4471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00: No bootstrap required, opened a new log
I20260620 02:03:55.213469 4471 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c38084da9df43a2bae44581a93a1f00" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33699 } }
I20260620 02:03:55.213936 4471 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:03:55.214037 4471 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c38084da9df43a2bae44581a93a1f00, State: Initialized, Role: FOLLOWER
I20260620 02:03:55.214850 4471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c38084da9df43a2bae44581a93a1f00" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33699 } }
I20260620 02:03:55.215193 4471 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:03:55.215382 4471 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:03:55.215610 4471 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:03:55.223197 4471 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c38084da9df43a2bae44581a93a1f00" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33699 } }
I20260620 02:03:55.225759 4471 leader_election.cc:304] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c38084da9df43a2bae44581a93a1f00; no voters:
I20260620 02:03:55.226711 4471 leader_election.cc:290] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:03:55.226977 4476 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:03:55.228729 4476 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [term 1 LEADER]: Becoming Leader. State: Replica: 5c38084da9df43a2bae44581a93a1f00, State: Running, Role: LEADER
I20260620 02:03:55.229769 4471 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:03:55.230657 4476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c38084da9df43a2bae44581a93a1f00" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33699 } }
I20260620 02:03:55.239542 4478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c38084da9df43a2bae44581a93a1f00. Latest consensus state: current_term: 1 leader_uuid: "5c38084da9df43a2bae44581a93a1f00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c38084da9df43a2bae44581a93a1f00" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33699 } } }
I20260620 02:03:55.239677 4477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c38084da9df43a2bae44581a93a1f00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c38084da9df43a2bae44581a93a1f00" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33699 } } }
I20260620 02:03:55.240236 4478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:55.240242 4477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00 [sys.catalog]: This master's current role is: LEADER
I20260620 02:03:55.276253 4487 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:03:55.298424 4487 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:03:55.322814 4487 catalog_manager.cc:1382] Generated new cluster ID: f2e4cd806a6949d5848acec1aa5f78cc
I20260620 02:03:55.322984 4487 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:03:55.354933 4487 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:03:55.357908 4487 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:03:55.377168 4487 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 5c38084da9df43a2bae44581a93a1f00: Generated new TSK 0
I20260620 02:03:55.378511 4487 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:55.500322 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.107s user 0.005s sys 0.036s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:03:55.650313 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.150s user 0.003s sys 0.035s
I20260620 02:03:55.651705 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:33699
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:40771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:03:56.239480 4499 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:03:56.239936 4499 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:03:56.240195 4499 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:03:56.252267 4499 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:03:56.252542 4499 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:03:56.269054 4499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:33699
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:03:56.270964 4499 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:03:56.272655 4499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:03:56.289009 4507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:56.289191 4504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:03:56.289139 4505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:03:56.291229 4499 server_base.cc:1061] running on GCE node
I20260620 02:03:56.292145 4499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:03:56.293737 4499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:03:56.295238 4499 hybrid_clock.cc:648] HybridClock initialized: now 1781921036295126 us; error 102 us; skew 500 ppm
I20260620 02:03:56.295897 4499 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:03:56 dist-test-slave-wg14 krb5kdc[4387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921036, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:03:56.304029 4499 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:03:56.306612 4499 webserver.cc:533] Webserver started at http://127.20.1.129:37907/ using document root <none> and password file <none>
I20260620 02:03:56.307668 4499 fs_manager.cc:362] Metadata directory not provided
I20260620 02:03:56.307855 4499 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:03:56.308357 4499 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:03:56.311568 4499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "189fc61ca9044d228d2c17272d8f3529"
format_stamp: "Formatted at 2026-06-20 02:03:56 on dist-test-slave-wg14"
I20260620 02:03:56.312700 4499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "189fc61ca9044d228d2c17272d8f3529"
format_stamp: "Formatted at 2026-06-20 02:03:56 on dist-test-slave-wg14"
I20260620 02:03:56.323814 4499 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s
I20260620 02:03:56.330874 4514 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:56.332978 4499 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:03:56.333253 4499 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "189fc61ca9044d228d2c17272d8f3529"
format_stamp: "Formatted at 2026-06-20 02:03:56 on dist-test-slave-wg14"
I20260620 02:03:56.333534 4499 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:03:56.367028 4499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:03:56.373452 4499 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:03:56.373819 4499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:03:56.375119 4499 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:03:56.378062 4499 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:03:56.378224 4499 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:56.378386 4499 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:03:56.378456 4499 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:03:56.432088 4499 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:35089
I20260620 02:03:56.432183 4627 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:35089 every 8 connection(s)
I20260620 02:03:56.433835 4499 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:03:56.440972 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 4499
I20260620 02:03:56.441177 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 02:03:56 dist-test-slave-wg14 krb5kdc[4387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921036, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:03:56.475457 4628 heartbeater.cc:344] Connected to a master server at 127.20.1.190:33699
I20260620 02:03:56.476035 4628 heartbeater.cc:461] Registering TS with master...
I20260620 02:03:56.477315 4628 heartbeater.cc:507] Master 127.20.1.190:33699 requested a full tablet report, sending...
I20260620 02:03:56.483614 4435 ts_manager.cc:194] Registered new tserver with Master: 189fc61ca9044d228d2c17272d8f3529 (127.20.1.129:35089)
I20260620 02:03:56.487069 4435 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:45805
I20260620 02:03:56.490026 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Jun 20 02:03:56 dist-test-slave-wg14 krb5kdc[4387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:03:56 dist-test-slave-wg14 krb5kdc[4387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.20.1.190@KRBTEST.COM
I20260620 02:03:56.571085 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4499
Jun 20 02:03:56 dist-test-slave-wg14 krb5kdc[4387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:03:56.594375 4623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:56.755604 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 4499
I20260620 02:03:56.783264 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4403
I20260620 02:03:56.785804 4466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:03:56.949266 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 4403
2026-06-20T02:03:56Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/9 (6278 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/10
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:03:57 dist-test-slave-wg14 krb5kdc[4641](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:03:57 dist-test-slave-wg14 krb5kdc[4641](info): set up 2 sockets
Jun 20 02:03:57 dist-test-slave-wg14 krb5kdc[4641](info): commencing operation
krb5kdc: starting...
W20260620 02:03:59.817236 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.813s user 0.001s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 02:04:00 dist-test-slave-wg14 krb5kdc[4641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:04:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:04:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:04:00.316679 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.190: real 0.118s user 0.008s sys 0.019s
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
I20260620 02:04:00.409204 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:46699
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:46699
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:04:01.205123 4657 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:04:01.206862 4657 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:04:01.207103 4657 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:04:01.227331 4657 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:04:01.227478 4657 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:04:01.227540 4657 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:04:01.227581 4657 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:04:01.257033 4657 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:46699
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:46699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:04:01.259656 4657 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:04:01.262388 4657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:04:01.282210 4663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:04:01.282106 4662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:04:01.285252 4665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:04:01.287348 4657 server_base.cc:1061] running on GCE node
I20260620 02:04:01.288810 4657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:04:01.293113 4657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:04:01.295904 4657 hybrid_clock.cc:648] HybridClock initialized: now 1781921041295744 us; error 123 us; skew 500 ppm
I20260620 02:04:01.296887 4657 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:04:01 dist-test-slave-wg14 krb5kdc[4641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921041, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:04:01.302042 4657 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:04:01.305459 4657 webserver.cc:533] Webserver started at http://127.20.1.190:34307/ using document root <none> and password file <none>
I20260620 02:04:01.308054 4657 fs_manager.cc:362] Metadata directory not provided
I20260620 02:04:01.310271 4657 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:04:01.311843 4657 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:04:01.315470 4657 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "0640261fc4dc4553a2c5299a99992d9b"
format_stamp: "Formatted at 2026-06-20 02:04:01 on dist-test-slave-wg14"
I20260620 02:04:01.317301 4657 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "0640261fc4dc4553a2c5299a99992d9b"
format_stamp: "Formatted at 2026-06-20 02:04:01 on dist-test-slave-wg14"
I20260620 02:04:01.325373 4657 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260620 02:04:01.337484 4673 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:04:01.339761 4657 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.004s
I20260620 02:04:01.340040 4657 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "0640261fc4dc4553a2c5299a99992d9b"
format_stamp: "Formatted at 2026-06-20 02:04:01 on dist-test-slave-wg14"
I20260620 02:04:01.340363 4657 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:04:01.373610 4657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:04:01.389009 4657 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:04:01.390846 4657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:04:01.440030 4657 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:46699
I20260620 02:04:01.440093 4724 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:46699 every 8 connection(s)
I20260620 02:04:01.443584 4657 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:04:01.448693 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 4657
I20260620 02:04:01.448952 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:04:01.453382 4725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:04:01.471361 4725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b: Bootstrap starting.
I20260620 02:04:01.476451 4725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b: Neither blocks nor log segments found. Creating new log.
I20260620 02:04:01.479961 4725 log.cc:826] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:04:01 dist-test-slave-wg14 krb5kdc[4641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:04:01.486229 4725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b: No bootstrap required, opened a new log
I20260620 02:04:01.501472 4725 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0640261fc4dc4553a2c5299a99992d9b" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46699 } }
I20260620 02:04:01.502275 4725 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:04:01.503895 4725 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0640261fc4dc4553a2c5299a99992d9b, State: Initialized, Role: FOLLOWER
I20260620 02:04:01.505414 4725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0640261fc4dc4553a2c5299a99992d9b" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46699 } }
I20260620 02:04:01.506330 4725 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:04:01.507619 4725 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:04:01.508119 4725 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:04:01.512744 4725 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0640261fc4dc4553a2c5299a99992d9b" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46699 } }
I20260620 02:04:01.514222 4725 leader_election.cc:304] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0640261fc4dc4553a2c5299a99992d9b; no voters:
I20260620 02:04:01.517428 4725 leader_election.cc:290] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:04:01.517469 4730 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:04:01.518575 4730 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [term 1 LEADER]: Becoming Leader. State: Replica: 0640261fc4dc4553a2c5299a99992d9b, State: Running, Role: LEADER
I20260620 02:04:01.520056 4730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0640261fc4dc4553a2c5299a99992d9b" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46699 } }
I20260620 02:04:01.522483 4725 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:04:01.525669 4732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0640261fc4dc4553a2c5299a99992d9b. Latest consensus state: current_term: 1 leader_uuid: "0640261fc4dc4553a2c5299a99992d9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0640261fc4dc4553a2c5299a99992d9b" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46699 } } }
I20260620 02:04:01.525612 4731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0640261fc4dc4553a2c5299a99992d9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0640261fc4dc4553a2c5299a99992d9b" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 46699 } } }
I20260620 02:04:01.527175 4731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [sys.catalog]: This master's current role is: LEADER
I20260620 02:04:01.528483 4732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b [sys.catalog]: This master's current role is: LEADER
I20260620 02:04:01.535755 4737 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:04:01.547434 4737 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:04:01.570334 4737 catalog_manager.cc:1382] Generated new cluster ID: ac06cf8e19c74f5a812ed7f194931667
I20260620 02:04:01.571024 4737 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:04:01.647235 4737 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:04:01.656683 4737 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:04:01.709445 4737 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 0640261fc4dc4553a2c5299a99992d9b: Generated new TSK 0
I20260620 02:04:01.710479 4737 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:04:01.860896 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.138s user 0.000s sys 0.032s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:04:01.985858 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.125s user 0.002s sys 0.049s
I20260620 02:04:01.987118 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:46699
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:04:02.704542 4753 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:04:02.704950 4753 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:04:02.705078 4753 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:04:02.725992 4753 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:04:02.731477 4753 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:04:02.746768 4753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:46699
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:04:02.749032 4753 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:04:02.757629 4753 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:04:02.781541 4758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:04:02.782552 4759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:04:02.782357 4761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:04:02.785005 4753 server_base.cc:1061] running on GCE node
I20260620 02:04:02.788012 4753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:04:02.792641 4753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:04:02.794442 4753 hybrid_clock.cc:648] HybridClock initialized: now 1781921042794321 us; error 95 us; skew 500 ppm
I20260620 02:04:02.795053 4753 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:04:02 dist-test-slave-wg14 krb5kdc[4641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921042, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:04:02.801388 4753 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:04:02.808436 4753 webserver.cc:533] Webserver started at http://127.20.1.129:42991/ using document root <none> and password file <none>
I20260620 02:04:02.809755 4753 fs_manager.cc:362] Metadata directory not provided
I20260620 02:04:02.809927 4753 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:04:02.811463 4753 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:04:02.818040 4753 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "823961a88c4943f9ba6e1e5cff89dc25"
format_stamp: "Formatted at 2026-06-20 02:04:02 on dist-test-slave-wg14"
I20260620 02:04:02.819387 4753 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "823961a88c4943f9ba6e1e5cff89dc25"
format_stamp: "Formatted at 2026-06-20 02:04:02 on dist-test-slave-wg14"
I20260620 02:04:02.835878 4753 fs_manager.cc:696] Time spent creating directory manager: real 0.015s user 0.014s sys 0.000s
I20260620 02:04:02.840974 4768 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:04:02.844120 4753 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:04:02.844485 4753 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "823961a88c4943f9ba6e1e5cff89dc25"
format_stamp: "Formatted at 2026-06-20 02:04:02 on dist-test-slave-wg14"
I20260620 02:04:02.844995 4753 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:04:02.898655 4753 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:04:02.905576 4753 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:04:02.906744 4753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:04:02.908231 4753 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:04:02.910943 4753 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:04:02.911095 4753 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:04:02.911307 4753 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:04:02.911383 4753 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:04:02.999812 4753 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:43651
I20260620 02:04:02.999827 4881 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:43651 every 8 connection(s)
I20260620 02:04:03.001942 4753 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:04:03.002976 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 4753
I20260620 02:04:03.003408 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 02:04:03 dist-test-slave-wg14 krb5kdc[4641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921042, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:04:03.056726 4882 heartbeater.cc:344] Connected to a master server at 127.20.1.190:46699
I20260620 02:04:03.057922 4882 heartbeater.cc:461] Registering TS with master...
I20260620 02:04:03.059305 4882 heartbeater.cc:507] Master 127.20.1.190:46699 requested a full tablet report, sending...
I20260620 02:04:03.063242 4689 ts_manager.cc:194] Registered new tserver with Master: 823961a88c4943f9ba6e1e5cff89dc25 (127.20.1.129:43651)
I20260620 02:04:03.064797 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260620 02:04:03.065603 4689 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:34481
Jun 20 02:04:03 dist-test-slave-wg14 krb5kdc[4641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:04:03 dist-test-slave-wg14 krb5kdc[4641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.20.1.190@KRBTEST.COM
I20260620 02:04:03.168886 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4753
Jun 20 02:04:03 dist-test-slave-wg14 krb5kdc[4641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:04:03.202282 4877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:04:03.379365 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 4753
I20260620 02:04:03.397106 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4657
I20260620 02:04:03.398558 4720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:04:03.551837 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 4657
2026-06-20T02:04:03Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/10 (6597 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/11
Loading random data
Initializing database '/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:04:03 dist-test-slave-wg14 krb5kdc[4895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:04:03 dist-test-slave-wg14 krb5kdc[4895](info): set up 2 sockets
Jun 20 02:04:03 dist-test-slave-wg14 krb5kdc[4895](info): commencing operation
krb5kdc: starting...
W20260620 02:04:06.256386 20486 mini_kdc.cc:121] Time spent starting KDC: real 2.654s user 0.002s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/test-user.keytab.
Jun 20 02:04:06 dist-test-slave-wg14 krb5kdc[4895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:04:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:04:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:04:06.800993 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.190: real 0.120s user 0.001s sys 0.025s
WARNING: no policy specified for HTTP/127.20.1.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:04:06.918177 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.190: real 0.117s user 0.000s sys 0.030s
I20260620 02:04:06.919417 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:33267
--webserver_interface=127.20.1.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:33267
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:04:07.710439 4911 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:04:07.712273 4911 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:04:07.712391 4911 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:04:07.723661 4911 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:04:07.723805 4911 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:04:07.723867 4911 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:04:07.723909 4911 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:04:07.743700 4911 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:33267
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:33267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:04:07.745675 4911 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:04:07.754230 4911 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:04:07.773244 4917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:04:07.773422 4916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:04:07.779525 4920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:04:07.780197 4911 server_base.cc:1061] running on GCE node
I20260620 02:04:07.782078 4911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:04:07.785267 4911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:04:07.787528 4911 hybrid_clock.cc:648] HybridClock initialized: now 1781921047787428 us; error 81 us; skew 500 ppm
I20260620 02:04:07.788239 4911 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:04:07 dist-test-slave-wg14 krb5kdc[4895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921047, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:04:07.795745 4911 init.cc:377] Logged in from keytab as kudu/127.20.1.190@KRBTEST.COM (short username kudu)
I20260620 02:04:07.800639 4911 webserver.cc:533] Webserver started at http://127.20.1.190:42053/ using document root <none> and password file <none>
I20260620 02:04:07.802924 4911 fs_manager.cc:362] Metadata directory not provided
I20260620 02:04:07.803481 4911 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:04:07.804095 4911 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:04:07.808971 4911 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "ee49896eaec04e4ebd04cf219ccbbd88"
format_stamp: "Formatted at 2026-06-20 02:04:07 on dist-test-slave-wg14"
I20260620 02:04:07.810354 4911 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "ee49896eaec04e4ebd04cf219ccbbd88"
format_stamp: "Formatted at 2026-06-20 02:04:07 on dist-test-slave-wg14"
I20260620 02:04:07.823802 4911 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.005s sys 0.004s
I20260620 02:04:07.835837 4926 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:04:07.839280 4911 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.009s sys 0.000s
I20260620 02:04:07.839607 4911 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "ee49896eaec04e4ebd04cf219ccbbd88"
format_stamp: "Formatted at 2026-06-20 02:04:07 on dist-test-slave-wg14"
I20260620 02:04:07.840022 4911 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:04:07.892338 4911 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:04:07.902459 4911 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:04:07.902863 4911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:04:07.943249 4911 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:33267
I20260620 02:04:07.943322 4978 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:33267 every 8 connection(s)
I20260620 02:04:07.945369 4911 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:04:07.952304 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 4911
I20260620 02:04:07.952605 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:04:07.959319 4979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:04:07 dist-test-slave-wg14 krb5kdc[4895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:04:07.983345 4979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88: Bootstrap starting.
I20260620 02:04:07.990211 4979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88: Neither blocks nor log segments found. Creating new log.
I20260620 02:04:07.996349 4979 log.cc:826] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88: Log is configured to *not* fsync() on all Append() calls
I20260620 02:04:08.003696 4979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88: No bootstrap required, opened a new log
I20260620 02:04:08.015056 4979 raft_consensus.cc:359] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee49896eaec04e4ebd04cf219ccbbd88" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33267 } }
I20260620 02:04:08.016158 4979 raft_consensus.cc:385] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:04:08.016311 4979 raft_consensus.cc:740] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee49896eaec04e4ebd04cf219ccbbd88, State: Initialized, Role: FOLLOWER
I20260620 02:04:08.017639 4979 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee49896eaec04e4ebd04cf219ccbbd88" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33267 } }
I20260620 02:04:08.018026 4979 raft_consensus.cc:399] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:04:08.018198 4979 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:04:08.018407 4979 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:04:08.021142 4979 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee49896eaec04e4ebd04cf219ccbbd88" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33267 } }
I20260620 02:04:08.022210 4979 leader_election.cc:304] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee49896eaec04e4ebd04cf219ccbbd88; no voters:
I20260620 02:04:08.024313 4979 leader_election.cc:290] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:04:08.024791 4984 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:04:08.026176 4984 raft_consensus.cc:697] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [term 1 LEADER]: Becoming Leader. State: Replica: ee49896eaec04e4ebd04cf219ccbbd88, State: Running, Role: LEADER
I20260620 02:04:08.028040 4984 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee49896eaec04e4ebd04cf219ccbbd88" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33267 } }
I20260620 02:04:08.033855 4979 sys_catalog.cc:565] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:04:08.036969 4985 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee49896eaec04e4ebd04cf219ccbbd88. Latest consensus state: current_term: 1 leader_uuid: "ee49896eaec04e4ebd04cf219ccbbd88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee49896eaec04e4ebd04cf219ccbbd88" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33267 } } }
I20260620 02:04:08.037127 4986 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee49896eaec04e4ebd04cf219ccbbd88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee49896eaec04e4ebd04cf219ccbbd88" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 33267 } } }
I20260620 02:04:08.037676 4985 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [sys.catalog]: This master's current role is: LEADER
I20260620 02:04:08.037683 4986 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88 [sys.catalog]: This master's current role is: LEADER
I20260620 02:04:08.040992 4989 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:04:08.055209 4989 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:04:08.083981 4989 catalog_manager.cc:1382] Generated new cluster ID: 120a74652b0e4b25b6cb6948e798a46e
I20260620 02:04:08.084151 4989 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:04:08.119796 4989 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:04:08.122340 4989 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:04:08.160239 4989 catalog_manager.cc:6069] T 00000000000000000000000000000000 P ee49896eaec04e4ebd04cf219ccbbd88: Generated new TSK 0
I20260620 02:04:08.161401 4989 catalog_manager.cc:1549] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:04:08.320505 20486 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.1.129: real 0.137s user 0.003s sys 0.030s
WARNING: no policy specified for HTTP/127.20.1.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.1.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.1.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/kudu.keytab.
W20260620 02:04:08.445032 20486 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.1.129: real 0.124s user 0.000s sys 0.035s
I20260620 02:04:08.445938 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.1.129:0
--local_ip_for_outbound_sockets=127.20.1.129
--webserver_interface=127.20.1.129
--webserver_port=0
--tserver_master_addrs=127.20.1.190:33267
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.1.148:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:04:09.251624 5007 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:04:09.253253 5007 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:04:09.253465 5007 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:04:09.270143 5007 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:04:09.272101 5007 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.1.129
I20260620 02:04:09.299520 5007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.1.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.1.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.1.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.1.190:33267
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.1.129
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
I20260620 02:04:09.302441 5007 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:04:09.304330 5007 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260620 02:04:09.331259 5007 server_base.cc:1061] running on GCE node
W20260620 02:04:09.331195 5012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:04:09.331368 5013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:04:09.331342 5015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:04:09.333602 5007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:04:09.337037 5007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:04:09.338548 5007 hybrid_clock.cc:648] HybridClock initialized: now 1781921049338488 us; error 50 us; skew 500 ppm
I20260620 02:04:09.339275 5007 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:04:09 dist-test-slave-wg14 krb5kdc[4895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921049, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:04:09.344257 5007 init.cc:377] Logged in from keytab as kudu/127.20.1.129@KRBTEST.COM (short username kudu)
I20260620 02:04:09.347045 5007 webserver.cc:533] Webserver started at http://127.20.1.129:37933/ using document root <none> and password file <none>
I20260620 02:04:09.352383 5007 fs_manager.cc:362] Metadata directory not provided
I20260620 02:04:09.352941 5007 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:04:09.354151 5007 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:04:09.359675 5007 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/ts-0/data/instance:
uuid: "15e367c0af9043c1a8d5a757dfdee66b"
format_stamp: "Formatted at 2026-06-20 02:04:09 on dist-test-slave-wg14"
I20260620 02:04:09.360880 5007 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance:
uuid: "15e367c0af9043c1a8d5a757dfdee66b"
format_stamp: "Formatted at 2026-06-20 02:04:09 on dist-test-slave-wg14"
I20260620 02:04:09.373001 5007 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.007s sys 0.001s
I20260620 02:04:09.378001 5022 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:04:09.380400 5007 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 02:04:09.380664 5007 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
uuid: "15e367c0af9043c1a8d5a757dfdee66b"
format_stamp: "Formatted at 2026-06-20 02:04:09 on dist-test-slave-wg14"
I20260620 02:04:09.380921 5007 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:04:09.418674 5007 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:04:09.429092 5007 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:04:09.429525 5007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:04:09.433665 5007 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 02:04:09.437279 5007 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:04:09.437846 5007 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260620 02:04:09.437999 5007 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 02:04:09.438053 5007 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:04:09.511425 5007 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.129:32779
I20260620 02:04:09.511469 5134 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.129:32779 every 8 connection(s)
I20260620 02:04:09.513314 5007 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/ts-0/data/info.pb
I20260620 02:04:09.523324 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 5007
I20260620 02:04:09.523536 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1781920181147249-20486-0/minicluster-data/ts-0/wal/instance
Jun 20 02:04:09 dist-test-slave-wg14 krb5kdc[4895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921049, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.1.129@KRBTEST.COM for kudu/127.20.1.190@KRBTEST.COM
I20260620 02:04:09.553478 5135 heartbeater.cc:344] Connected to a master server at 127.20.1.190:33267
I20260620 02:04:09.554006 5135 heartbeater.cc:461] Registering TS with master...
I20260620 02:04:09.555282 5135 heartbeater.cc:507] Master 127.20.1.190:33267 requested a full tablet report, sending...
I20260620 02:04:09.558990 4943 ts_manager.cc:194] Registered new tserver with Master: 15e367c0af9043c1a8d5a757dfdee66b (127.20.1.129:32779)
I20260620 02:04:09.562057 4943 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.1.129@KRBTEST.COM'} at 127.20.1.129:35367
I20260620 02:04:09.566459 20486 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Jun 20 02:04:09 dist-test-slave-wg14 krb5kdc[4895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:04:09 dist-test-slave-wg14 krb5kdc[4895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.20.1.190@KRBTEST.COM
I20260620 02:04:09.661394 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 5007
Jun 20 02:04:09 dist-test-slave-wg14 krb5kdc[4895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781921049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.1.129@KRBTEST.COM
I20260620 02:04:09.692539 5130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:04:09.815275 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 5007
I20260620 02:04:09.838639 20486 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu pid 4911
I20260620 02:04:09.840243 4974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:04:09.982569 20486 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu with pid 4911
2026-06-20T02:04:10Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/11 (6435 ms)
[----------] 37 tests from ExternalMiniClusterTest (868787 ms total)
[----------] 1 test from Kudu3762Test
[ RUN ] Kudu3762Test.AddMaster
2026-06-20T02:04:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:04:10Z Disabled control of system clock
I20260620 02:04:10.096685 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42533
--webserver_interface=127.20.1.190
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:42533,127.20.1.189:45373
--flush_threshold_mb=0
--flush_threshold_secs=1
--log_cache_size_limit_mb=1
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_segment_size_mb=1
--maintenance_manager_num_threads=4
--maintenance_manager_polling_interval_ms=10
--unlock_experimental_flags=true with env {}
W20260620 02:04:10.634316 5149 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:04:10.634747 5149 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:04:10.634908 5149 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:04:10.651204 5149 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20260620 02:04:10.651377 5149 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20260620 02:04:10.651508 5149 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:04:10.651556 5149 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:04:10.651629 5149 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:04:10.651714 5149 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:04:10.651835 5149 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260620 02:04:10.651891 5149 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
I20260620 02:04:10.668519 5149 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42533,127.20.1.189:45373
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.190:42533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.1.190
--webserver_port=0
--flush_threshold_mb=0
--flush_threshold_secs=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_num_threads=4
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
W20260620 02:04:10.668929 5149 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.20.1.190:42533,127.20.1.189:45373'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 02:04:10.670610 5149 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:04:10.672968 5149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:04:10.690618 5155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:04:10.690650 5154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:04:10.691563 5157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:04:10.693171 5149 server_base.cc:1061] running on GCE node
I20260620 02:04:10.694160 5149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:04:10.696051 5149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:04:10.699687 5149 hybrid_clock.cc:648] HybridClock initialized: now 1781921050699529 us; error 55 us; skew 500 ppm
I20260620 02:04:10.700815 5149 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:04:10.705103 5149 webserver.cc:533] Webserver started at http://127.20.1.190:38755/ using document root <none> and password file <none>
I20260620 02:04:10.706226 5149 fs_manager.cc:362] Metadata directory not provided
I20260620 02:04:10.706429 5149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:04:10.706941 5149 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:04:10.713965 5149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-0/data/instance:
uuid: "62f4b9d35d704ffe8ad6afae0a24f077"
format_stamp: "Formatted at 2026-06-20 02:04:10 on dist-test-slave-wg14"
I20260620 02:04:10.716727 5149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-0/wal/instance:
uuid: "62f4b9d35d704ffe8ad6afae0a24f077"
format_stamp: "Formatted at 2026-06-20 02:04:10 on dist-test-slave-wg14"
I20260620 02:04:10.725932 5149 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260620 02:04:10.736539 5164 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:04:10.738910 5149 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:04:10.739270 5149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-0/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-0/wal
uuid: "62f4b9d35d704ffe8ad6afae0a24f077"
format_stamp: "Formatted at 2026-06-20 02:04:10 on dist-test-slave-wg14"
I20260620 02:04:10.739576 5149 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:04:10.785779 5149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:04:10.792085 5149 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:04:10.792560 5149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:04:10.838742 5149 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.190:42533
I20260620 02:04:10.838786 5220 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.190:42533 every 8 connection(s)
I20260620 02:04:10.840942 5149 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-0/data/info.pb
I20260620 02:04:10.841923 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 5149
I20260620 02:04:10.842144 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-0/wal/instance
I20260620 02:04:10.843997 20486 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
/tmp/dist-test-taskpZv5rS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:45373
--webserver_interface=127.20.1.189
--webserver_port=0
--builtin_ntp_servers=127.20.1.148:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.1.190:42533,127.20.1.189:45373
--flush_threshold_mb=0
--flush_threshold_secs=1
--log_cache_size_limit_mb=1
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_segment_size_mb=1
--maintenance_manager_num_threads=4
--maintenance_manager_polling_interval_ms=10
--unlock_experimental_flags=true with env {}
I20260620 02:04:10.859020 5221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:04:10.867925 5221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42533 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:04:10.902644 5221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } has no permanent_uuid. Determining permanent_uuid...
W20260620 02:04:10.904970 5165 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45373: connect: Connection refused (error 111)
W20260620 02:04:10.913188 5221 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:45373: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45373: connect: Connection refused (error 111)
I20260620 02:04:10.963692 5221 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } attempt: 1
W20260620 02:04:10.973462 5221 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:45373: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45373: connect: Connection refused (error 111)
I20260620 02:04:11.041971 5221 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } attempt: 2
W20260620 02:04:11.045195 5221 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:45373: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45373: connect: Connection refused (error 111)
I20260620 02:04:11.136830 5221 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } attempt: 3
W20260620 02:04:11.141309 5221 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:45373: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45373: connect: Connection refused (error 111)
I20260620 02:04:11.285780 5221 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } attempt: 4
W20260620 02:04:11.290586 5221 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:45373: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45373: connect: Connection refused (error 111)
W20260620 02:04:11.427881 5223 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:04:11.428308 5223 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:04:11.428445 5223 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260620 02:04:11.441632 5223 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20260620 02:04:11.442113 5223 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20260620 02:04:11.442581 5223 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:04:11.442814 5223 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:04:11.442942 5223 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:04:11.443082 5223 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:04:11.443326 5223 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260620 02:04:11.443727 5223 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
I20260620 02:04:11.460114 5223 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.1.148:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.1.190:42533,127.20.1.189:45373
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.1.189:45373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.1.189
--webserver_port=0
--flush_threshold_mb=0
--flush_threshold_secs=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_num_threads=4
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision faec34f53d4c9dd1e3a6dd2b7314d2fc7aadff6c
build type FASTDEBUG
built by None at 20 Jun 2026 01:43:21 UTC on bdcb31816ec0
build id 12478
ASAN enabled
W20260620 02:04:11.460602 5223 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.20.1.190:42533,127.20.1.189:45373'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 02:04:11.463622 5223 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:04:11.468154 5223 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 02:04:11.486356 5234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:04:11.487108 5235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:04:11.491171 5237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:04:11.494792 5223 server_base.cc:1061] running on GCE node
I20260620 02:04:11.498997 5223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:04:11.505002 5223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:04:11.506439 5223 hybrid_clock.cc:648] HybridClock initialized: now 1781921051506319 us; error 143 us; skew 500 ppm
I20260620 02:04:11.507030 5223 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:04:11.511286 5223 webserver.cc:533] Webserver started at http://127.20.1.189:39085/ using document root <none> and password file <none>
I20260620 02:04:11.512703 5223 fs_manager.cc:362] Metadata directory not provided
I20260620 02:04:11.512944 5223 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:04:11.513470 5223 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:04:11.520840 5223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-1/data/instance:
uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137"
format_stamp: "Formatted at 2026-06-20 02:04:11 on dist-test-slave-wg14"
I20260620 02:04:11.523104 5223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-1/wal/instance:
uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137"
format_stamp: "Formatted at 2026-06-20 02:04:11 on dist-test-slave-wg14"
I20260620 02:04:11.540751 5223 fs_manager.cc:696] Time spent creating directory manager: real 0.017s user 0.015s sys 0.000s
I20260620 02:04:11.548532 5243 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:04:11.551589 5223 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:04:11.551913 5223 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-1/data,/tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-1/wal
uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137"
format_stamp: "Formatted at 2026-06-20 02:04:11 on dist-test-slave-wg14"
I20260620 02:04:11.552254 5223 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:04:11.588466 5223 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 02:04:11.589918 5223 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:04:11.589967 5221 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } attempt: 5
I20260620 02:04:11.590409 5223 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260620 02:04:11.593025 5221 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.1.189:45373: Network error: Client connection negotiation failed: client connection to 127.20.1.189:45373: connect: Connection refused (error 111)
I20260620 02:04:11.635041 5223 rpc_server.cc:307] RPC server started. Bound to: 127.20.1.189:45373
I20260620 02:04:11.635041 5298 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.1.189:45373 every 8 connection(s)
I20260620 02:04:11.637986 5223 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-1/data/info.pb
I20260620 02:04:11.642238 20486 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpZv5rS/build/asan/bin/kudu as pid 5223
I20260620 02:04:11.642473 20486 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpZv5rS/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1781920181147249-20486-0/minicluster-data/master-1/wal/instance
I20260620 02:04:11.648758 5299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:04:11.667786 5299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42533 } has no permanent_uuid. Determining permanent_uuid...
************************ BEGIN STACKS **************************
I20260620 02:04:11.690284 5299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:04:11.705708 5299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137: Bootstrap starting.
I20260620 02:04:11.710666 5299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137: Neither blocks nor log segments found. Creating new log.
I20260620 02:04:11.714206 5299 log.cc:826] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137: Log is configured to *not* fsync() on all Append() calls
I20260620 02:04:11.720539 5299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137: No bootstrap required, opened a new log
I20260620 02:04:11.731029 5299 raft_consensus.cc:359] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f4b9d35d704ffe8ad6afae0a24f077" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42533 } } peers { permanent_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } }
I20260620 02:04:11.732170 5299 raft_consensus.cc:385] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:04:11.732354 5299 raft_consensus.cc:740] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d607aaa2a4fc4e3eaa2a7f46d46fa137, State: Initialized, Role: FOLLOWER
I20260620 02:04:11.733286 5299 consensus_queue.cc:260] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f4b9d35d704ffe8ad6afae0a24f077" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42533 } } peers { permanent_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } }
I20260620 02:04:11.734604 5307 sys_catalog.cc:455] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f4b9d35d704ffe8ad6afae0a24f077" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42533 } } peers { permanent_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } } }
I20260620 02:04:11.735098 5307 sys_catalog.cc:458] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:04:11.736266 5299 sys_catalog.cc:565] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:04:11.739573 5252 tablet.cc:2406] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:04:11.758303 5318 catalog_manager.cc:1593] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:04:11.760047 5318 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:04:11.753513 5223 master_runner.cc:187] Error getting master registration for 127.20.1.190:42533: OK, instance_id { permanent_uuid: "62f4b9d35d704ffe8ad6afae0a24f077" instance_seqno: 1781921050823053 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:04:11.767644 5223 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:04:11.849637 5307 raft_consensus.cc:493] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:04:11.850116 5307 raft_consensus.cc:515] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f4b9d35d704ffe8ad6afae0a24f077" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42533 } } peers { permanent_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } }
I20260620 02:04:11.855098 5307 leader_election.cc:290] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62f4b9d35d704ffe8ad6afae0a24f077 (127.20.1.190:42533)
I20260620 02:04:12.140367 5221 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } attempt: 6
I20260620 02:04:12.175154 5221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077: Bootstrap starting.
I20260620 02:04:12.188022 5221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077: Neither blocks nor log segments found. Creating new log.
I20260620 02:04:12.190735 5221 log.cc:826] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077: Log is configured to *not* fsync() on all Append() calls
I20260620 02:04:12.204205 5221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077: No bootstrap required, opened a new log
I20260620 02:04:12.216255 5221 raft_consensus.cc:359] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f4b9d35d704ffe8ad6afae0a24f077" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42533 } } peers { permanent_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } }
I20260620 02:04:12.217082 5221 raft_consensus.cc:385] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:04:12.217319 5221 raft_consensus.cc:740] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f4b9d35d704ffe8ad6afae0a24f077, State: Initialized, Role: FOLLOWER
I20260620 02:04:12.219266 5221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f4b9d35d704ffe8ad6afae0a24f077" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42533 } } peers { permanent_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } }
I20260620 02:04:12.223356 5221 sys_catalog.cc:565] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:04:12.223943 5196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f4b9d35d704ffe8ad6afae0a24f077" is_pre_election: true
I20260620 02:04:12.227308 5196 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d607aaa2a4fc4e3eaa2a7f46d46fa137 in term 0.
I20260620 02:04:12.229503 5244 leader_election.cc:304] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 62f4b9d35d704ffe8ad6afae0a24f077, d607aaa2a4fc4e3eaa2a7f46d46fa137; no voters:
I20260620 02:04:12.230480 5307 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:04:12.230720 5307 raft_consensus.cc:493] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:04:12.230870 5307 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:04:12.234849 5307 raft_consensus.cc:515] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f4b9d35d704ffe8ad6afae0a24f077" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42533 } } peers { permanent_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } }
W20260620 02:04:12.235495 5174 tablet.cc:2406] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:04:12.238191 5307 leader_election.cc:290] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [CANDIDATE]: Term 1 election: Requested vote from peers 62f4b9d35d704ffe8ad6afae0a24f077 (127.20.1.190:42533)
I20260620 02:04:12.239498 5195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f4b9d35d704ffe8ad6afae0a24f077"
I20260620 02:04:12.246189 5195 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:04:12.255118 5195 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d607aaa2a4fc4e3eaa2a7f46d46fa137 in term 1.
I20260620 02:04:12.255986 5244 leader_election.cc:304] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 62f4b9d35d704ffe8ad6afae0a24f077, d607aaa2a4fc4e3eaa2a7f46d46fa137; no voters:
I20260620 02:04:12.256567 5307 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:04:12.240084 5323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f4b9d35d704ffe8ad6afae0a24f077" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42533 } } peers { permanent_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } } }
I20260620 02:04:12.257974 5323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:04:12.258927 5307 raft_consensus.cc:697] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [term 1 LEADER]: Becoming Leader. State: Replica: d607aaa2a4fc4e3eaa2a7f46d46fa137, State: Running, Role: LEADER
I20260620 02:04:12.261513 5307 consensus_queue.cc:237] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f4b9d35d704ffe8ad6afae0a24f077" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42533 } } peers { permanent_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } }
I20260620 02:04:12.292177 5325 sys_catalog.cc:455] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d607aaa2a4fc4e3eaa2a7f46d46fa137. Latest consensus state: current_term: 1 leader_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f4b9d35d704ffe8ad6afae0a24f077" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42533 } } peers { permanent_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } } }
I20260620 02:04:12.292910 5325 sys_catalog.cc:458] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [sys.catalog]: This master's current role is: LEADER
I20260620 02:04:12.304891 5331 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 02:04:12.308113 5331 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 02:04:12.336170 5195 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077 [term 1 FOLLOWER]: Refusing update from remote peer d607aaa2a4fc4e3eaa2a7f46d46fa137: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:04:12.338855 5325 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f4b9d35d704ffe8ad6afae0a24f077" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:04:12.345281 5338 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:04:12.365408 5323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d607aaa2a4fc4e3eaa2a7f46d46fa137. Latest consensus state: current_term: 1 leader_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f4b9d35d704ffe8ad6afae0a24f077" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42533 } } peers { permanent_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } } }
I20260620 02:04:12.365729 5323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:04:12.371099 5334 mvcc.cc:204] Tried to move back new op lower bound from 7298748630330933248 to 7298748630092107776. Current Snapshot: MvccSnapshot[applied={T|T < 7298748630330933248}]
I20260620 02:04:12.373697 5325 sys_catalog.cc:455] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f4b9d35d704ffe8ad6afae0a24f077" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42533 } } peers { permanent_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } } }
I20260620 02:04:12.374074 5325 sys_catalog.cc:458] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137 [sys.catalog]: This master's current role is: LEADER
I20260620 02:04:12.378106 5323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f4b9d35d704ffe8ad6afae0a24f077" member_type: VOTER last_known_addr { host: "127.20.1.190" port: 42533 } } peers { permanent_uuid: "d607aaa2a4fc4e3eaa2a7f46d46fa137" member_type: VOTER last_known_addr { host: "127.20.1.189" port: 45373 } } }
I20260620 02:04:12.378412 5323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:04:12.384006 5252 maintenance_manager.cc:419] P d607aaa2a4fc4e3eaa2a7f46d46fa137: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260620 02:04:12.385983 5331 catalog_manager.cc:1382] Generated new cluster ID: d36405ace4da46ea85079c0affb41abb
I20260620 02:04:12.386214 5331 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 02:04:12.388665 5174 maintenance_manager.cc:419] P 62f4b9d35d704ffe8ad6afae0a24f077: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260620 02:04:12.445730 5331 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 02:04:12.459230 5331 catalog_manager.cc:1539] Loading token signing keys...
I20260620 02:04:12.555614 5174 maintenance_manager.cc:419] P 62f4b9d35d704ffe8ad6afae0a24f077: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260620 02:04:12.556915 5174 maintenance_manager.cc:419] P 62f4b9d35d704ffe8ad6afae0a24f077: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260620 02:04:12.554934 5173 maintenance_manager.cc:643] P 62f4b9d35d704ffe8ad6afae0a24f077: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.165s user 0.049s sys 0.029s Metrics: {"bytes_written":4776,"cfile_init":1,"compiler_manager_pool.queue_time_us":7753,"dirs.queue_time_us":17493,"dirs.run_cpu_time_us":313,"dirs.run_wall_time_us":1244,"drs_written":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1543,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":1,"thread_start_us":14701,"threads_started":2}
I20260620 02:04:12.559175 5171 maintenance_manager.cc:643] P 62f4b9d35d704ffe8ad6afae0a24f077: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20260620 02:04:12.591583 5172 maintenance_manager.cc:643] P 62f4b9d35d704ffe8ad6afae0a24f077: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.035s user 0.018s sys 0.009s Metrics: {"bytes_written":5619,"cfile_init":1,"dirs.queue_time_us":88,"dirs.run_cpu_time_us":736,"dirs.run_wall_time_us":3826,"drs_written":1,"lbm_read_time_us":177,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10190,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":26,"peak_mem_usage":0,"rows_written":1,"spinlock_wait_cycles":1280}
I20260620 02:04:12.601589 5331 catalog_manager.cc:6069] T 00000000000000000000000000000000 P d607aaa2a4fc4e3eaa2a7f46d46fa137: Generated new TSK 0
I20260620 02:04:12.601763 5174 maintenance_manager.cc:419] P 62f4b9d35d704ffe8ad6afae0a24f077: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260620 02:04:12.602411 5174 maintenance_manager.cc:419] P 62f4b9d35d704ffe8ad6afae0a24f077: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260620 02:04:12.602872 5174 maintenance_manager.cc:419] P 62f4b9d35d704ffe8ad6afae0a24f077: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260620 02:04:12.603031 5331 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 02:04:12.602912 5172 maintenance_manager.cc:643] P 62f4b9d35d704ffe8ad6afae0a24f077: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":238,"lbm_reads_lt_1ms":4}
I20260620 02:04:12.604233 5173 maintenance_manager.cc:643] P 62f4b9d35d704ffe8ad6afae0a24f077: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260620 02:04:12.604595 5172 maintenance_manager.cc:643] P 62f4b9d35d704ffe8ad6afae0a24f077: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260620 02:04:12.605013 5251 maintenance_manager.cc:643] P d607aaa2a4fc4e3eaa2a7f46d46fa137: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.215s user 0.055s sys 0.013s Metrics: {"bytes_written":4776,"cfile_init":1,"compiler_manager_pool.queue_time_us":161626,"compiler_manager_pool.run_cpu_time_us":69,"compiler_manager_pool.run_wall_time_us":75,"dirs.queue_time_us":14173,"dirs.run_cpu_time_us":298,"dirs.run_wall_time_us":8152,"drs_written":1,"lbm_read_time_us":558,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1077,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":1,"thread_start_us":14406,"threads_started":1}
I20260620 02:04:12.608129 5252 maintenance_manager.cc:419] P d607aaa2a4fc4e3eaa2a7f46d46fa137: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260620 02:04:12.612525 5252 maintenance_manager.cc:419] P d607aaa2a4fc4e3eaa2a7f46d46fa137: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260620 02:04:12.615350 5252 maintenance_manager.cc:419] P d607aaa2a4fc4e3eaa2a7f46d46fa137: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
[New LWP 20489]
[New LWP 20490]
[New LWP 5146]
I20260620 02:04:12.616745 5174 maintenance_manager.cc:419] P 62f4b9d35d704ffe8ad6afae0a24f077: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260620 02:04:12.618870 5251 maintenance_manager.cc:643] P d607aaa2a4fc4e3eaa2a7f46d46fa137: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
I20260620 02:04:12.620157 5249 maintenance_manager.cc:643] P d607aaa2a4fc4e3eaa2a7f46d46fa137: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260620 02:04:12.648546 5171 maintenance_manager.cc:643] P 62f4b9d35d704ffe8ad6afae0a24f077: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.031s user 0.020s sys 0.007s Metrics: {"bytes_written":5067,"cfile_init":1,"dirs.queue_time_us":113,"dirs.run_cpu_time_us":317,"dirs.run_wall_time_us":2370,"drs_written":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1721,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":1}
I20260620 02:04:12.648806 5174 maintenance_manager.cc:419] P 62f4b9d35d704ffe8ad6afae0a24f077: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260620 02:04:12.649441 5250 maintenance_manager.cc:643] P d607aaa2a4fc4e3eaa2a7f46d46fa137: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.036s user 0.024s sys 0.000s Metrics: {"bytes_written":5967,"cfile_init":1,"dirs.queue_time_us":128,"dirs.run_cpu_time_us":345,"dirs.run_wall_time_us":2922,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4236,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":25,"peak_mem_usage":0,"rows_written":2}
I20260620 02:04:12.649873 5174 maintenance_manager.cc:419] P 62f4b9d35d704ffe8ad6afae0a24f077: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260620 02:04:12.650246 5174 maintenance_manager.cc:419] P 62f4b9d35d704ffe8ad6afae0a24f077: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260620 02:04:12.650691 5172 maintenance_manager.cc:643] P 62f4b9d35d704ffe8ad6afae0a24f077: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
I20260620 02:04:12.651504 5252 maintenance_manager.cc:419] P d607aaa2a4fc4e3eaa2a7f46d46fa137: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.014997
I20260620 02:04:12.651680 5172 maintenance_manager.cc:643] P 62f4b9d35d704ffe8ad6afae0a24f077: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260620 02:04:12.651012 5173 maintenance_manager.cc:643] P 62f4b9d35d704ffe8ad6afae0a24f077: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260620 02:04:12.650669 5174 maintenance_manager.cc:419] P 62f4b9d35d704ffe8ad6afae0a24f077: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260620 02:04:12.658311 5173 maintenance_manager.cc:643] P 62f4b9d35d704ffe8ad6afae0a24f077: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260620 02:04:12.668743 5252 maintenance_manager.cc:419] P d607aaa2a4fc4e3eaa2a7f46d46fa137: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 174 bytes on disk
I20260620 02:04:12.676347 5250 maintenance_manager.cc:643] P d607aaa2a4fc4e3eaa2a7f46d46fa137: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260620 02:04:12.710448 5252 maintenance_manager.cc:419] P d607aaa2a4fc4e3eaa2a7f46d46fa137: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260620 02:04:12.710882 5252 maintenance_manager.cc:419] P d607aaa2a4fc4e3eaa2a7f46d46fa137: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260620 02:04:12.711256 5252 maintenance_manager.cc:419] P d607aaa2a4fc4e3eaa2a7f46d46fa137: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260620 02:04:12.710882 5249 maintenance_manager.cc:643] P d607aaa2a4fc4e3eaa2a7f46d46fa137: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.053s user 0.029s sys 0.013s Metrics: {"bytes_written":6027,"cfile_cache_hit":12,"cfile_cache_hit_bytes":1553,"cfile_cache_miss":28,"cfile_cache_miss_bytes":2195,"cfile_init":8,"delta_iterators_relevant":4,"dirs.queue_time_us":2943,"dirs.run_cpu_time_us":1441,"dirs.run_wall_time_us":12347,"drs_written":1,"lbm_read_time_us":1545,"lbm_reads_lt_1ms":60,"lbm_write_time_us":1632,"lbm_writes_lt_1ms":27,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":3,"thread_start_us":2582,"threads_started":4}
I20260620 02:04:12.711645 5250 maintenance_manager.cc:643] P d607aaa2a4fc4e3eaa2a7f46d46fa137: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20260620 02:04:12.711747 5248 maintenance_manager.cc:643] P d607aaa2a4fc4e3eaa2a7f46d46fa137: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260620 02:04:12.712009 5250 maintenance_manager.cc:643] P d607aaa2a4fc4e3eaa2a7f46d46fa137: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260620 02:04:13.364387 5338 catalog_manager.cc:1294] Loaded cluster ID: d36405ace4da46ea85079c0affb41abb
I20260620 02:04:13.364573 5338 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077: loading cluster ID for follower catalog manager: success
I20260620 02:04:13.371285 5338 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077: acquiring CA information for follower catalog manager: success
I20260620 02:04:13.380884 5338 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 62f4b9d35d704ffe8ad6afae0a24f077: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f7da4c6ead3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Id Target Id Frame
* 1 Thread 0x7f7d98d84200 (LWP 20486) "external_mini_c" 0x00007f7da4c6ead3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7f7d948bd700 (LWP 20489) "external_mini_c" 0x00007f7da4c7332a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7f7d928c4700 (LWP 20490) "kernel-watcher-" 0x00007f7da4c6efb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
4 Thread 0x7f7d920c3700 (LWP 5146) "rpc reactor-514" 0x00007f7d9991d947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7f7d920c3700 (LWP 5146)):
#0 0x00007f7d9991d947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f7d9b80e25d in epoll_poll (loop=0x61a000304e80, timeout=0.022238600997752656) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f7d9b812ba3 in ev_run (loop=0x61a000304e80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f7da30c8b61 in ev::loop_ref::run (this=this@entry=0x6150005fba18, flags=671232, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f7da30bc6e6 in kudu::rpc::ReactorThread::RunThread (this=0x6150005fba10) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:611
#5 0x00007f7da30c17bc in kudu::rpc::ReactorThread::Init()::$_2::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:299
#6 0x00007f7da30c1639 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_2>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x000000000049fe3d in std::function<void ()>::operator()() const (this=0x6120003e7098) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007f7d9d934f49 in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007f7da4c686db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f7d9991d61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7f7d928c4700 (LWP 20490)):
#0 0x00007f7da4c6efb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f7d9d6e9f5e in kudu::ConditionVariable::WaitUntil (this=0x612000020398, until=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/condition_variable.cc:87
#2 0x00007f7db248b35c in kudu::CountDownLatch::WaitUntil (this=<optimized out>, when=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/countdown_latch.h:89
#3 0x00007f7db248314a in kudu::CountDownLatch::WaitFor (this=0x6120000203c0, delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/countdown_latch.h:99
#4 0x00007f7d9d7f264f in kudu::KernelStackWatchdog::RunThread (this=0x6120000202c0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/kernel_stack_watchdog.cc:134
#5 0x00007f7d9d7f361c in kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/kernel_stack_watchdog.cc:71
#6 0x00007f7d9d7f3499 in std::_Function_handler<void (), kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x000000000049fe3d in std::function<void ()>::operator()() const (this=0x612000020498) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007f7d9d934f49 in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007f7da4c686db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f7d9991d61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 2 (Thread 0x7f7d948bd700 (LWP 20489)):
#0 0x00007f7da4c7332a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000003a2c4f in __interceptor_waitpid (pid=<optimized out>, status=0x7f7d948bb710, options=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2434
#2 0x00007f7d9d91e705 in kudu::Subprocess::DoWait (this=<optimized out>, wait_status=<optimized out>, mode=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/subprocess.cc:849
#3 0x00007f7d9d91e385 in kudu::Subprocess::Wait (this=0x7f7d948bb710, wait_status=0x0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/subprocess.cc:561
#4 0x00007f7d9d7d69ee in kudu::PstackWatcher::RunStackDump (argv=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/pstack_watcher.cc:260
#5 0x00007f7d9d7d6274 in kudu::PstackWatcher::RunGdbStackDump (pid=<optimized out>, flags=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/pstack_watcher.cc:218
#6 0x00007f7d9d7d5709 in kudu::PstackWatcher::DumpPidStacks (pid=<optimized out>, flags=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/pstack_watcher.cc:169
#7 0x00007f7d9d7d3a97 in kudu::PstackWatcher::DumpStacks (flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/pstack_watcher.cc:162
#8 0x000000000049e97b in kudu::CreateAndStartTimeoutThread()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:66
#9 0x000000000049e813 in _ZSt13__invoke_implIvZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEET_St14__invoke_otherOT0_DpOT1_ (__f=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:60
#10 0x000000000049e793 in _ZSt8__invokeIZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS3_DpOS4_ (__fn=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
#11 0x000000000049e769 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::_M_invoke<0ul> (this=this@entry=0x602000012118) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
#12 0x000000000049e729 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::operator()() (this=this@entry=0x602000012118) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
#13 0x000000000049e62b in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> > >::_M_run() (this=<optimized out>) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
#14 0x00007f7d99ec26df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007f7da4c686db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007f7d9991d61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f7d98d84200 (LWP 20486)):
#0 0x00007f7da4c6ead3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f7d9d6e9c37 in kudu::ConditionVariable::Wait (this=0x60e000201b70) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/condition_variable.cc:55
#2 0x00007f7db34345f9 in kudu::CountDownLatch::Wait (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/countdown_latch.h:79
#3 0x00007f7db2ecc6cb in kudu::Synchronizer::Wait (this=this@entry=0x7ffc78c6c9c0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/async_util.h:58
#4 0x00007f7db2ea876b in kudu::cluster::ExternalMiniCluster::GetLeaderMasterIndex (this=0x6190009d5380, idx=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:1083
#5 0x00007f7db2ea1d3b in kudu::cluster::ExternalMiniCluster::leader_master (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:1047
#6 0x00007f7db2e93603 in kudu::cluster::ExternalMiniCluster::AddTabletServer (this=0x6190009d5380) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:656
#7 0x00007f7db2e8e8cf in kudu::cluster::ExternalMiniCluster::Start (this=0x6190009d5380) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:471
#8 0x000000000045061f in kudu::cluster::Kudu3762Test_AddMaster_Test::TestBody (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster-test.cc:420
#9 0x00007f7d9e2400ed in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7f7d9e24182b "the test body", method=<optimized out>, object=0x6070001224a0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#10 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x6070001224a0, method=<optimized out>, location=location@entry=0x7f7d9e24182b "the test body") at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#11 0x00007f7d9e234bdb in testing::Test::Run (this=0x6070001224a0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674
#12 0x00007f7d9e234d9d in testing::TestInfo::Run (this=0x6120000010c0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853
#13 0x00007f7d9e235377 in testing::TestSuite::Run (this=0x612000001240) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012
#14 0x00007f7d9e23577c in testing::internal::UnitTestImpl::RunAllTests (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870
#15 0x00007f7d9e24060d in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7f7d9e2433e8 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x616000005a80) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#16 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x616000005a80, method=<optimized out>, location=location@entry=0x7f7d9e2433e8 "auxiliary test code (environments or event listeners)") at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#17 0x00007f7d9e234e63 in testing::UnitTest::Run (this=0x7f7d9e4541e0 <testing::UnitTest::GetInstance()::instance>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444
#18 0x000000000049fd42 in RUN_ALL_TESTS () at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2293
#19 0x000000000049dc19 in main (argc=<optimized out>, argv=0x7ffc78c717b8) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:115
W20260620 02:04:55.979471 5146 connection.cc:256] RPC call timeout handler was delayed by -39.738786118000007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.40309945300000005s for a response
F20260620 02:04:56.021238 20486 external_mini_cluster.cc:656] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1781921096 (unix time) try "date -d @1781921096" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800005006) received by PID 20486 (TID 0x7f7d98d84200) from PID 20486; stack trace: ***
@ 0x7f7da4c73980 (unknown) at ??:0
@ 0x7f7d9983ae87 gsignal at ??:0
@ 0x7f7d9983c7f1 abort at ??:0
@ 0x7f7d9c302dcd google::LogMessage::Fail() at ??:0
@ 0x7f7d9c306b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f7d9c3027cc google::LogMessage::Flush() at ??:0
@ 0x7f7d9c303f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7db2e95306 kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7f7db2e8e8cf kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x45061f kudu::cluster::Kudu3762Test_AddMaster_Test::TestBody() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster-test.cc:420
@ 0x7f7d9e2400ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f7d9e234bdb testing::Test::Run() at ??:0
@ 0x7f7d9e234d9d testing::TestInfo::Run() at ??:0
@ 0x7f7d9e235377 testing::TestSuite::Run() at ??:0
@ 0x7f7d9e23577c testing::internal::UnitTestImpl::RunAllTests() at ??:0
************************* END STACKS ***************************
F20260620 02:04:56.072525 20489 test_main.cc:69] Maximum unit test time exceeded (870 sec)
*** Check failure stack trace: ***
@ 0x7f7d9e24060d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f7d9e234e63 testing::UnitTest::Run() at ??:0
@ 0x49fd42 RUN_ALL_TESTS() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2293
@ 0x49dc19 main at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:115
@ 0x7f7d9981dc87 __libc_start_main at ??:0
@ 0x36abba _start at ??:?